[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:15 dist-test-slave-gfsd krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:15 dist-test-slave-gfsd krb5kdc[399](info): set up 2 sockets
Sep 29 18:58:15 dist-test-slave-gfsd krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:58:16.790197 392 mini_kdc.cc:121] Time spent starting KDC: real 2.803s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 18:58:16.957302 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.167s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/test-user.keytab.
W20260929 18:58:17.945649 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.802s user 0.000s sys 0.001s
Sep 29 18:58:17 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260929 18:58:18.133841 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.166s user 0.002s sys 0.000s
I20260929 18:58:18.134285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:18.215173 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:18.215438 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:18.215479 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:18.216650 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:18.216723 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:18.216754 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:18.216790 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:18.216837 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:18.218309 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:18.218608 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:18.218860 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:18.223908 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.223954 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.224067 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.224161 412 server_base.cc:1034] running on GCE node
I20260929 18:58:18.224421 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:18.224717 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:18.225852 412 hybrid_clock.cc:648] HybridClock initialized: now 1790708298225833 us; error 32 us; skew 500 ppm
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:18.228327 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:18.228827 412 webserver.cc:460] Webserver started at http://127.0.98.62:37091/ using document root <none> and password file <none>
I20260929 18:58:18.229080 412 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:18.229151 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:18.229296 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:18.230327 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "d41f08c132594c9883212a0aac632918"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "25259570896e129a14dfec92d29d2e8c"
server_key_iv: "e214b6e46a6a023d3bbaf01e6599f045"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.230698 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "d41f08c132594c9883212a0aac632918"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "25259570896e129a14dfec92d29d2e8c"
server_key_iv: "e214b6e46a6a023d3bbaf01e6599f045"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.232167 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:18.232983 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.233151 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.233255 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "d41f08c132594c9883212a0aac632918"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "25259570896e129a14dfec92d29d2e8c"
server_key_iv: "e214b6e46a6a023d3bbaf01e6599f045"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.233327 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:18.265055 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:18.266996 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:18.267179 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:18.271229 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36391
I20260929 18:58:18.271317 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36391 every 8 connection(s)
I20260929 18:58:18.271672 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:18.272552 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:18.275017 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918: Bootstrap starting.
I20260929 18:58:18.275662 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:18.275967 480 log.cc:824] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:18.276695 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918: No bootstrap required, opened a new log
I20260929 18:58:18.278251 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260929 18:58:18.278414 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:18.278471 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41f08c132594c9883212a0aac632918, State: Initialized, Role: FOLLOWER
I20260929 18:58:18.278628 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260929 18:58:18.278692 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:18.278731 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:18.278803 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:18.279187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 412
I20260929 18:58:18.279315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:18.279438 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260929 18:58:18.279523 480 leader_election.cc:302] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41f08c132594c9883212a0aac632918; no voters:
I20260929 18:58:18.279699 480 leader_election.cc:288] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:18.279742 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:18.279877 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [term 1 LEADER]: Becoming Leader. State: Replica: d41f08c132594c9883212a0aac632918, State: Running, Role: LEADER
I20260929 18:58:18.279953 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:18.280031 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260929 18:58:18.280166 392 external_mini_cluster.cc:1428] Setting key 0f0fbf5aa34438b03ef5c6b8f8b704a6
I20260929 18:58:18.280375 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41f08c132594c9883212a0aac632918. Latest consensus state: current_term: 1 leader_uuid: "d41f08c132594c9883212a0aac632918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } } }
I20260929 18:58:18.280457 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:18.280658 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41f08c132594c9883212a0aac632918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41f08c132594c9883212a0aac632918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } } }
I20260929 18:58:18.280865 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:18.280664 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:18.281497 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:18.283560 490 catalog_manager.cc:1347] Generated new cluster ID: d17d4fa67a3f496882dfef36e554f1c7
I20260929 18:58:18.283615 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:18.295064 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.282074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.98.62:36391)
0929 18:58:18.282546 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.282554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.282587 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:58:18.284033 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.284037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.284079 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.284206 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.285623 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.286571 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.287679 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.288082 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.288201 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.291488 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.291492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.291493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.291510 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.293197 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.293641 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.293648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.293649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.293725 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.293990 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.293991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.293991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.294218 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.294388 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.294658 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.294790 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":70,"threads_started":1}
I20260929 18:58:18.295724 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:18.296389 490 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:18.313319 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41f08c132594c9883212a0aac632918: Generated new TSK 0
I20260929 18:58:18.313508 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:18.338272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:18.417912 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:18.418140 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:18.418181 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:18.419389 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:18.419446 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:18.419493 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:18.421090 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:18.421373 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:18.421597 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:18.426496 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.426494 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.426566 506 server_base.cc:1034] running on GCE node
W20260929 18:58:18.426542 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.426965 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:18.427198 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:18.428375 506 hybrid_clock.cc:648] HybridClock initialized: now 1790708298428366 us; error 45 us; skew 500 ppm
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:18.430897 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:18.431463 506 webserver.cc:460] Webserver started at http://127.0.98.1:46421/ using document root <none> and password file <none>
I20260929 18:58:18.431696 506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:18.431766 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:18.431910 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:18.432999 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e6df87d4d8642458d33a65b5e258069"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "77feb324c46b1def0a6924cf850fbef5"
server_key_iv: "16a306b74faa2bf704b0be888018059b"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.433409 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e6df87d4d8642458d33a65b5e258069"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "77feb324c46b1def0a6924cf850fbef5"
server_key_iv: "16a306b74faa2bf704b0be888018059b"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.434805 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:18.435586 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.435758 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:18.435877 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "8e6df87d4d8642458d33a65b5e258069"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "77feb324c46b1def0a6924cf850fbef5"
server_key_iv: "16a306b74faa2bf704b0be888018059b"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.435982 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:18.451222 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:18.453174 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:18.453282 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:18.453474 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:18.453711 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:18.453742 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.453763 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:18.453774 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.460080 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44299
I20260929 18:58:18.460134 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44299 every 8 connection(s)
I20260929 18:58:18.460449 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:18.462900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 506
I20260929 18:58:18.463006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:18.463158 392 external_mini_cluster.cc:1428] Setting key 5dd4990eee4137c520430ee5af2594df
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:18.470647 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.461153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52229 (local address 127.0.98.62:36391)
0929 18:58:18.461222 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.461224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.461426 (+ 202us) server_negotiation.cc:407] Connection header received
0929 18:58:18.461958 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.461959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.461974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.462024 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.462974 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.463741 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.464728 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.464943 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.465027 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.467393 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.467394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.467395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.467404 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.469113 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.469543 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.469545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.469545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.469606 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.469948 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.469949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.469950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.470087 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.470162 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.470440 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.470531 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:58:18.470667 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.461137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.1:52229)
0929 18:58:18.461307 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:18.461347 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:18.461846 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:18.462095 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:18.462098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:18.462653 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:18.462875 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.462879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.463817 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.463818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.464625 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.464626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.465036 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.465037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.465156 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.465684 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:18.465694 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:18.467266 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:18.469213 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.469218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.469220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.469463 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.469714 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.469715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.469715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.469870 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.470176 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:18.470182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:18.470295 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:58:18.470390 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:58:18.470507 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260929 18:58:18.471195 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260929 18:58:18.471338 635 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:18.471541 635 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260929 18:58:18.472072 444 ts_manager.cc:194] Registered new tserver with Master: 8e6df87d4d8642458d33a65b5e258069 (127.0.98.1:44299)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:58:18.472702 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:18.490105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:18.569079 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:18.569273 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:18.569295 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:18.570436 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:18.570482 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:18.570508 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:18.571949 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:18.572175 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:18.572417 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:18.577746 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.577775 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.577781 640 server_base.cc:1034] running on GCE node
W20260929 18:58:18.577751 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.578243 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:18.578501 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:18.579664 640 hybrid_clock.cc:648] HybridClock initialized: now 1790708298579641 us; error 43 us; skew 500 ppm
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:18.582744 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:18.583341 640 webserver.cc:460] Webserver started at http://127.0.98.2:45011/ using document root <none> and password file <none>
I20260929 18:58:18.583580 640 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:18.583626 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:18.583770 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:18.584892 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "1658e1f83037457bb107b2000ba6791c"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "357c1b8464defb43a9cdba250293b621"
server_key_iv: "743f6956ee2d877b7dea8d1deedfa7e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.585284 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1658e1f83037457bb107b2000ba6791c"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "357c1b8464defb43a9cdba250293b621"
server_key_iv: "743f6956ee2d877b7dea8d1deedfa7e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.586726 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:18.587774 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.588024 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:18.588135 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "1658e1f83037457bb107b2000ba6791c"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "357c1b8464defb43a9cdba250293b621"
server_key_iv: "743f6956ee2d877b7dea8d1deedfa7e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.588209 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:18.599462 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:18.601385 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:18.601547 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:18.601763 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:18.602067 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:18.602123 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.602155 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:18.602185 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.608750 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32779
I20260929 18:58:18.608798 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32779 every 8 connection(s)
I20260929 18:58:18.609133 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:18.614579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 640
I20260929 18:58:18.614678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:18.614856 392 external_mini_cluster.cc:1428] Setting key 1f5631ae4ef4d16983e7900f28b99c0b
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:18.618685 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.609833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.2:54155)
0929 18:58:18.610010 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:18.610049 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:18.610526 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:18.610759 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:18.610762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:18.611380 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:18.611567 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.611572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.612260 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.612261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.613069 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.613071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.613455 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.613456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.613569 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.614011 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:18.614019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:18.615603 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:18.617271 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.617277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.617279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.617595 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.617805 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.617806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.617806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.617939 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.618220 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:18.618224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:18.618329 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 18:58:18.618413 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:58:18.618534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260929 18:58:18.618700 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.609862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54155 (local address 127.0.98.62:36391)
0929 18:58:18.609923 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.609925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.610121 (+ 196us) server_negotiation.cc:407] Connection header received
0929 18:58:18.610633 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.610634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.610646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.610693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.611670 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.612164 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.613162 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.613367 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.613439 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.615715 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.615716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.615717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.615724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.617190 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.617676 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.617677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.617677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.617735 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.618009 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.618010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.618011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.618157 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.618226 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.618454 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.618564 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:58:18.619252 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260929 18:58:18.619379 769 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:18.619557 769 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260929 18:58:18.619976 444 ts_manager.cc:194] Registered new tserver with Master: 1658e1f83037457bb107b2000ba6791c (127.0.98.2:32779)
I20260929 18:58:18.620337 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54155
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:18.640069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:18.720316 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:18.720518 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:18.720541 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:18.721591 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:18.721623 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:18.721649 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:18.722981 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:18.723152 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:18.723328 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:18.728145 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.728161 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:18.728261 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:18.728467 774 server_base.cc:1034] running on GCE node
I20260929 18:58:18.728600 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:18.728778 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:18.729936 774 hybrid_clock.cc:648] HybridClock initialized: now 1790708298729932 us; error 36 us; skew 500 ppm
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:18.732401 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:18.732837 774 webserver.cc:460] Webserver started at http://127.0.98.3:39889/ using document root <none> and password file <none>
I20260929 18:58:18.733089 774 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:18.733135 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:18.733219 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:18.734202 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ec34d53e84e41bfadc78097db698960"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "e25e356f06ad22985670f8e192a40ec2"
server_key_iv: "d429d45f15e076fffc062ac515262331"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.734493 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ec34d53e84e41bfadc78097db698960"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "e25e356f06ad22985670f8e192a40ec2"
server_key_iv: "d429d45f15e076fffc062ac515262331"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.735895 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:18.736562 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.736747 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:18.736814 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "4ec34d53e84e41bfadc78097db698960"
format_stamp: "Formatted at 2026-09-29 18:58:18 on dist-test-slave-gfsd"
server_key: "e25e356f06ad22985670f8e192a40ec2"
server_key_iv: "d429d45f15e076fffc062ac515262331"
server_key_version: "encryptionkey@0"
I20260929 18:58:18.736907 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:18.750090 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:18.752384 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:18.752537 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:18.752746 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:18.753036 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:18.753089 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.753130 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:18.753158 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:18.760051 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34115
I20260929 18:58:18.760104 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34115 every 8 connection(s)
I20260929 18:58:18.760430 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:18.764381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 774
I20260929 18:58:18.764497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:18.764679 392 external_mini_cluster.cc:1428] Setting key c8741f452c8708b27c5ad2cbb88e24e8
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:18.769696 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.761177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39935 (local address 127.0.98.62:36391)
0929 18:58:18.761246 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.761248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.761531 (+ 283us) server_negotiation.cc:407] Connection header received
0929 18:58:18.762103 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.762104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.762115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.762157 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.763140 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.763644 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.764684 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.764958 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.765038 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.767181 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.767182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.767183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.767189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.768615 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.768966 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.768968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.768968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.769017 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.769226 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.769227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.769227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.769321 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.769380 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.769582 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.769635 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:58:18.769963 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.761158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.3:39935)
0929 18:58:18.761383 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:18.761422 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:18.761996 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:18.762218 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:18.762221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:18.762856 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:18.763050 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.763055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.763718 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.763718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.764589 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:18.764592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.765039 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.765040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:18.765191 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.765761 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:18.765770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:18.767101 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:18.768687 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.768692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.768694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.768892 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.769078 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:18.769079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:18.769079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:18.769171 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:18.769376 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:18.769380 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:18.769491 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 18:58:18.769571 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:58:18.769679 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260929 18:58:18.770533 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260929 18:58:18.770612 903 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:18.770745 903 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260929 18:58:18.771061 444 ts_manager.cc:194] Registered new tserver with Master: 4ec34d53e84e41bfadc78097db698960 (127.0.98.3:34115)
I20260929 18:58:18.771504 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39935
I20260929 18:58:18.776821 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:18.799891 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.791914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.0.98.1:44299)
0929 18:58:18.792115 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.792121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.792132 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:18.792165 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.792167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.792181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.792264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.792840 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.793603 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.794397 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.794728 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.794814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.797173 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.797183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.797183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.797198 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.798678 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.799050 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.799055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.799056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.799112 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.799335 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.799336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.799337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.799483 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.799645 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.799693 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.799771 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
W20260929 18:58:18.800272 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:18.822037 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:18.815005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.98.1:44299)
0929 18:58:18.815088 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:18.815090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:18.815193 (+ 103us) server_negotiation.cc:407] Connection header received
0929 18:58:18.815295 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:18.815296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:18.815309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:18.815367 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:18.815955 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.816432 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.817164 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:18.817376 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:18.817466 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:18.819711 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:18.819713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:18.819713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:18.819720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:18.821116 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.821434 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.821435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.821435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.821485 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:18.821647 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:18.821647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:18.821648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:18.821737 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:18.821823 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:18.821901 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:58:18.821973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:58:18.822594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 506
I20260929 18:58:18.827446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 640
I20260929 18:58:18.832952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 774
I20260929 18:58:18.837567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 412
2026-09-29T18:58:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4876 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[917](info): set up 2 sockets
Sep 29 18:58:18 dist-test-slave-gfsd krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 18:58:19.870813 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:19 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:19.990168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:20.068549 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:20.068749 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:20.068771 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:20.069936 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:20.069976 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:20.070011 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:20.070065 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:20.070088 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:20.071573 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:20.071820 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:20.072065 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:58:20.077098 930 server_base.cc:1034] running on GCE node
W20260929 18:58:20.077071 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.077044 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.077051 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:20.077494 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:20.077793 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:20.078974 930 hybrid_clock.cc:648] HybridClock initialized: now 1790708300078958 us; error 26 us; skew 500 ppm
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:20.081552 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:20.082062 930 webserver.cc:460] Webserver started at http://127.0.98.62:39561/ using document root <none> and password file <none>
I20260929 18:58:20.082290 930 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:20.082355 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:20.082504 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:20.083601 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "a04a006569a247f3937927137d20f1f5"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "da460053188f4fce0ec7e107e00ab0ce"
server_key_iv: "a65e85e5a67a06693ac50337b8a2ebb2"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.083961 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "a04a006569a247f3937927137d20f1f5"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "da460053188f4fce0ec7e107e00ab0ce"
server_key_iv: "a65e85e5a67a06693ac50337b8a2ebb2"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.085233 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:20.086047 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.086270 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:20.086391 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "a04a006569a247f3937927137d20f1f5"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "da460053188f4fce0ec7e107e00ab0ce"
server_key_iv: "a65e85e5a67a06693ac50337b8a2ebb2"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.086483 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:20.107988 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:20.110033 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:20.110198 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:20.114383 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46227
I20260929 18:58:20.114421 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46227 every 8 connection(s)
I20260929 18:58:20.114802 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:20.115742 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:20.117995 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5: Bootstrap starting.
I20260929 18:58:20.118510 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:20.118738 998 log.cc:824] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:20.119374 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5: No bootstrap required, opened a new log
I20260929 18:58:20.120914 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } }
I20260929 18:58:20.121042 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:20.121063 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a006569a247f3937927137d20f1f5, State: Initialized, Role: FOLLOWER
I20260929 18:58:20.121160 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } }
I20260929 18:58:20.121239 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:20.121273 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:20.121325 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:20.121807 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } }
I20260929 18:58:20.121930 998 leader_election.cc:302] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a04a006569a247f3937927137d20f1f5; no voters:
I20260929 18:58:20.122107 998 leader_election.cc:288] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:20.122157 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:20.122357 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:20.122550 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a006569a247f3937927137d20f1f5, State: Running, Role: LEADER
I20260929 18:58:20.122689 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } }
I20260929 18:58:20.123064 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a04a006569a247f3937927137d20f1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } } }
I20260929 18:58:20.123162 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:20.123312 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a04a006569a247f3937927137d20f1f5. Latest consensus state: current_term: 1 leader_uuid: "a04a006569a247f3937927137d20f1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a04a006569a247f3937927137d20f1f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46227 } } }
I20260929 18:58:20.123397 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:20.123670 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:20.124208 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:20.124876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 930
I20260929 18:58:20.124959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:20.125103 392 external_mini_cluster.cc:1428] Setting key f06c2a7932a565e424edcb2dca209ae4
I20260929 18:58:20.125594 1010 catalog_manager.cc:1347] Generated new cluster ID: 1876d284cc5d488786fe45d7e3ea89d9
I20260929 18:58:20.125648 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:20.136366 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.125557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.0.98.62:46227)
0929 18:58:20.125762 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.125771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.125805 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:58:20.125935 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.125941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.125991 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.126117 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.126856 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.128238 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.129300 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.129898 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.129993 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.132734 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.132738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.132738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.132749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.134732 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.135110 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.135113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.135114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.135219 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.135452 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.135453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.135453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.135762 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.135925 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.135998 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.136145 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260929 18:58:20.138043 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:20.138517 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:20.142741 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a04a006569a247f3937927137d20f1f5: Generated new TSK 0
I20260929 18:58:20.142987 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:20.170339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:20.272152 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:20.272346 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:20.272375 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:20.273617 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:20.273659 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:20.273686 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:20.275195 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:20.275475 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:20.275703 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:20.280725 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.280745 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.280906 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:20.281059 1024 server_base.cc:1034] running on GCE node
I20260929 18:58:20.281280 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:20.281526 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:20.282698 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790708300282680 us; error 35 us; skew 500 ppm
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:20.293584 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:20.294121 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34863/ using document root <none> and password file <none>
I20260929 18:58:20.294368 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:20.294436 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:20.294593 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:20.295804 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "fba39b5b92c14d2dac8f554b95e60725"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "665eb59b60c13c9d85f7c7218b5fd7ef"
server_key_iv: "4542cae366f48d96553fa0d2e7c28521"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.296166 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fba39b5b92c14d2dac8f554b95e60725"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "665eb59b60c13c9d85f7c7218b5fd7ef"
server_key_iv: "4542cae366f48d96553fa0d2e7c28521"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.297523 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 18:58:20.298331 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.298549 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:20.298657 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "fba39b5b92c14d2dac8f554b95e60725"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "665eb59b60c13c9d85f7c7218b5fd7ef"
server_key_iv: "4542cae366f48d96553fa0d2e7c28521"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.298735 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:20.325568 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:20.327742 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:20.327912 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:20.328158 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:20.328500 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:20.328555 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.328590 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:20.328639 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.334404 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41989
I20260929 18:58:20.334455 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41989 every 8 connection(s)
I20260929 18:58:20.334771 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:20.335786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1024
I20260929 18:58:20.335880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:20.336042 392 external_mini_cluster.cc:1428] Setting key 4c749fb14aeb16b7afdded0ba175fdc5
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:58:20.344650 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.335447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35587 (local address 127.0.98.62:46227)
0929 18:58:20.335489 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.335492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.335711 (+ 219us) server_negotiation.cc:407] Connection header received
0929 18:58:20.336413 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.336415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.336435 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.336504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.337486 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.337991 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.339011 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.339218 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.339318 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.341804 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.341805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.341806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.341812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.343238 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.343754 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.343755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.343755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.343805 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.344085 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.344085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.344086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.344190 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.344243 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.344521 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.344587 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:20.344784 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.335409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46227 (local address 127.0.98.1:35587)
0929 18:58:20.335566 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.335627 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.336241 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.336586 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.336590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.337183 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:20.337369 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.337374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.338101 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.338102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.338907 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.338909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.339315 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.339316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.339466 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.339997 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:20.340005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:20.341715 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:20.343372 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.343378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.343381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.343674 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.343869 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.343869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.343870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.344013 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.344261 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:20.344267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:20.344404 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.344491 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.344601 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 18:58:20.345352 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46227
I20260929 18:58:20.345465 1153 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:20.345638 1153 heartbeater.cc:511] Master 127.0.98.62:46227 requested a full tablet report, sending...
I20260929 18:58:20.346231 962 ts_manager.cc:194] Registered new tserver with Master: fba39b5b92c14d2dac8f554b95e60725 (127.0.98.1:41989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:58:20.347057 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:20.361773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:20.441375 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:20.441571 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:20.441594 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:20.442662 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:20.442695 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:20.442721 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:20.444147 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:20.444314 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:20.444474 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:58:20.449692 1158 server_base.cc:1034] running on GCE node
W20260929 18:58:20.449676 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.449667 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.449676 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:20.450166 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:20.450436 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:20.451596 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790708300451592 us; error 35 us; skew 500 ppm
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:20.454172 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:20.454612 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40585/ using document root <none> and password file <none>
I20260929 18:58:20.454840 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:20.454916 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:20.455065 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:20.456166 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "2fc701461b7245e0ba761cec2ab9b8d6"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "b83d5ad67bb561c41f19af66cd1e668f"
server_key_iv: "b6fdf77c9f55d105dce019a22893172e"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.456535 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2fc701461b7245e0ba761cec2ab9b8d6"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "b83d5ad67bb561c41f19af66cd1e668f"
server_key_iv: "b6fdf77c9f55d105dce019a22893172e"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.458360 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:58:20.459101 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.459331 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:20.459414 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "2fc701461b7245e0ba761cec2ab9b8d6"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "b83d5ad67bb561c41f19af66cd1e668f"
server_key_iv: "b6fdf77c9f55d105dce019a22893172e"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.459498 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:20.474035 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:20.476297 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:20.476462 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:20.476699 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:20.477011 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:20.477066 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.477109 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:20.477136 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.482758 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37319
I20260929 18:58:20.482811 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37319 every 8 connection(s)
I20260929 18:58:20.483120 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:20.485821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1158
I20260929 18:58:20.485903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:20.486064 392 external_mini_cluster.cc:1428] Setting key 921770fc519f4bee3533854ce7344ca5
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:20.493808 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.483861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49219 (local address 127.0.98.62:46227)
0929 18:58:20.483908 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.483910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.484210 (+ 300us) server_negotiation.cc:407] Connection header received
0929 18:58:20.484735 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.484737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.484749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.484795 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.485733 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.486357 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.487564 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.487894 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.487973 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.490595 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.490596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.490597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.490605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.492368 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.492770 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.492771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.492772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.492865 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.493110 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.493111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.493111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.493263 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.493343 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.493601 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.493685 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:20.493834 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.483909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46227 (local address 127.0.98.2:49219)
0929 18:58:20.484077 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.484111 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.484610 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.484875 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.484880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.485438 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:20.485622 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.485626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.486458 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.486458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.487487 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.487490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.487989 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.487990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.488146 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.488668 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:20.488679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:20.490496 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:20.492456 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.492462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.492464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.492688 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.492942 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.492943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.492943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.493042 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.493345 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:20.493349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:20.493464 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.493552 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.493675 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260929 18:58:20.494421 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46227
I20260929 18:58:20.494540 1287 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:20.494740 1287 heartbeater.cc:511] Master 127.0.98.62:46227 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:58:20.495153 962 ts_manager.cc:194] Registered new tserver with Master: 2fc701461b7245e0ba761cec2ab9b8d6 (127.0.98.2:37319)
I20260929 18:58:20.495563 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:20.512307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:20.597587 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:20.597779 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:20.597802 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:20.598886 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:20.598918 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:20.598944 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:20.600333 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:20.600520 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:20.600692 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:20.605577 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.605566 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:20.605808 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:20.606045 1292 server_base.cc:1034] running on GCE node
I20260929 18:58:20.606288 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:20.606523 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:20.607677 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790708300607674 us; error 38 us; skew 500 ppm
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:20.610028 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:20.610553 1292 webserver.cc:460] Webserver started at http://127.0.98.3:40465/ using document root <none> and password file <none>
I20260929 18:58:20.610790 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:20.610857 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:20.611004 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:20.612097 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "04327123a43544b69122e550fdbf3aec"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "6b1b7572b05f6e06255ee4a72a9e18a4"
server_key_iv: "f772ad4fd17f0357ff17fe562bd76773"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.612453 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04327123a43544b69122e550fdbf3aec"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "6b1b7572b05f6e06255ee4a72a9e18a4"
server_key_iv: "f772ad4fd17f0357ff17fe562bd76773"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.613708 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:58:20.614398 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.614596 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:20.614694 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "04327123a43544b69122e550fdbf3aec"
format_stamp: "Formatted at 2026-09-29 18:58:20 on dist-test-slave-gfsd"
server_key: "6b1b7572b05f6e06255ee4a72a9e18a4"
server_key_iv: "f772ad4fd17f0357ff17fe562bd76773"
server_key_version: "encryptionkey@0"
I20260929 18:58:20.614771 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:20.625288 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:20.627185 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:20.627363 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:20.627590 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:20.627871 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:20.627925 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.627964 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:20.627992 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:20.633826 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42793
I20260929 18:58:20.633874 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42793 every 8 connection(s)
I20260929 18:58:20.634155 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:20.636631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1292
I20260929 18:58:20.636705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:20.636860 392 external_mini_cluster.cc:1428] Setting key 41315f589a75442c0f74ce8d00b4328e
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:20.643092 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.634842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58551 (local address 127.0.98.62:46227)
0929 18:58:20.634911 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.634913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.635077 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:58:20.635593 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.635594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.635603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.635655 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.636500 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.637211 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.638376 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.638561 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.638617 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.640628 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.640629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.640629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.640635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.641822 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.642204 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.642205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.642205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.642252 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.642457 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.642457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.642458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.642549 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.642606 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.642968 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.643030 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:58:20.643146 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.634817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46227 (local address 127.0.98.3:58551)
0929 18:58:20.634960 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.634999 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.635510 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.635710 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.635713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.636228 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:20.636412 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.636416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.637314 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.637315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.638295 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.638297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.638633 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.638634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.638730 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.639118 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:20.639125 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:20.640547 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:20.641934 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.641939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.641941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.642141 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.642308 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:20.642308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:20.642308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:20.642403 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:20.642620 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:20.642625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:20.642789 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.642890 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.643003 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260929 18:58:20.643589 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46227
I20260929 18:58:20.643663 1421 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:20.643838 1421 heartbeater.cc:511] Master 127.0.98.62:46227 requested a full tablet report, sending...
I20260929 18:58:20.644157 962 ts_manager.cc:194] Registered new tserver with Master: 04327123a43544b69122e550fdbf3aec (127.0.98.3:42793)
I20260929 18:58:20.644526 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58551
I20260929 18:58:20.648720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:20.655676 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.650173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.0.98.62:46227)
0929 18:58:20.650235 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.650236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.650331 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:58:20.650397 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.650398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.650405 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.650454 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.650968 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.651452 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.652191 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.652363 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.652422 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.653565 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.653566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.653566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.653571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.654714 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.655022 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.655022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.655022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.655069 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.655214 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.655214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.655214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.655388 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.655462 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.655563 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.655619 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:58:20.657634 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:20.659243 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:20.666009 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.661580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42793 (local address 127.0.0.1:56784)
0929 18:58:20.661859 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.661900 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.661970 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.662295 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.662298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.662301 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:20.662448 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.662451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.663595 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.663596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.664769 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.664772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.665685 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.665772 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.665785 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.665859 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.665922 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":64,"threads_started":1}
I20260929 18:58:20.666039 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.661697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.98.3:42793)
0929 18:58:20.662142 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.662146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.662155 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:20.662191 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.662194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.662210 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.662276 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:20.662534 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.663481 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.664858 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.665593 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665670 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.665725 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.665894 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.665940 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":70,"threads_started":1}
I20260929 18:58:20.666347 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.661798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41989 (local address 127.0.0.1:41240)
0929 18:58:20.662407 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.662417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.662472 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.662681 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.662682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.662684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:20.662730 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.662731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.664029 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.664030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.665128 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.665131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.666112 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.666113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.666174 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.666179 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.666235 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.666277 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":64,"thread_start_us":78,"threads_started":1}
I20260929 18:58:20.666368 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.662010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.98.1:41989)
0929 18:58:20.662225 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.662231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.662538 (+ 307us) server_negotiation.cc:407] Connection header received
0929 18:58:20.662584 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.662587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.662600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.662678 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:20.662800 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.663908 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665228 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.665954 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.666022 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.666075 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.666253 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.666295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":145,"threads_started":1}
I20260929 18:58:20.666392 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.661776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.0.98.2:37319)
0929 18:58:20.662134 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.662141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.662349 (+ 208us) server_negotiation.cc:407] Connection header received
0929 18:58:20.662398 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.662401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.662417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.662492 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:20.662710 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.663908 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665138 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.665625 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665719 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.665783 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.666058 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.666308 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":70,"threads_started":1}
I20260929 18:58:20.666020 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.661626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37319 (local address 127.0.0.1:33916)
0929 18:58:20.662261 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.662272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.662332 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.662577 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.662578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.662580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:20.662632 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.662634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.664022 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.664023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.665056 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.665058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.665814 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.665815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.665876 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.665879 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.665923 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.665964 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260929 18:58:20.667003 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 1152ccb538094600a672419677d03687 (DEFAULT_TABLE table=test-table [id=d70acdf40d5347e2b1b511dd1f09caf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:20.667380 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1152ccb538094600a672419677d03687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:20.667420 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1152ccb538094600a672419677d03687 (DEFAULT_TABLE table=test-table [id=d70acdf40d5347e2b1b511dd1f09caf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:20.667690 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1152ccb538094600a672419677d03687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:20.667598 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1152ccb538094600a672419677d03687 (DEFAULT_TABLE table=test-table [id=d70acdf40d5347e2b1b511dd1f09caf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:20.667884 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1152ccb538094600a672419677d03687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:20.670082 1438 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: Bootstrap starting.
I20260929 18:58:20.670678 1438 tablet_bootstrap.cc:654] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:20.670656 1439 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: Bootstrap starting.
I20260929 18:58:20.671267 1438 log.cc:824] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:20.671379 1439 tablet_bootstrap.cc:654] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:20.671474 1440 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Bootstrap starting.
I20260929 18:58:20.671972 1439 log.cc:824] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:20.672098 1438 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: No bootstrap required, opened a new log
I20260929 18:58:20.672117 1440 tablet_bootstrap.cc:654] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:20.672171 1438 ts_tablet_manager.cc:1397] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:58:20.672638 1440 log.cc:824] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:20.672734 1439 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: No bootstrap required, opened a new log
I20260929 18:58:20.672802 1439 ts_tablet_manager.cc:1397] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:58:20.673445 1440 tablet_bootstrap.cc:492] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: No bootstrap required, opened a new log
I20260929 18:58:20.673499 1440 ts_tablet_manager.cc:1397] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:20.673898 1438 raft_consensus.cc:348] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.674031 1438 raft_consensus.cc:374] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:20.674052 1438 raft_consensus.cc:729] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba39b5b92c14d2dac8f554b95e60725, State: Initialized, Role: FOLLOWER
I20260929 18:58:20.674162 1438 consensus_queue.cc:260] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.674423 1438 ts_tablet_manager.cc:1428] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 18:58:20.674806 1153 heartbeater.cc:503] Master 127.0.98.62:46227 was elected leader, sending a full tablet report...
I20260929 18:58:20.675140 1439 raft_consensus.cc:348] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.675313 1439 raft_consensus.cc:374] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:20.675349 1439 raft_consensus.cc:729] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04327123a43544b69122e550fdbf3aec, State: Initialized, Role: FOLLOWER
I20260929 18:58:20.675598 1439 consensus_queue.cc:260] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.675863 1439 ts_tablet_manager.cc:1428] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:58:20.675880 1421 heartbeater.cc:503] Master 127.0.98.62:46227 was elected leader, sending a full tablet report...
I20260929 18:58:20.676257 1440 raft_consensus.cc:348] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.676410 1440 raft_consensus.cc:374] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:20.676455 1440 raft_consensus.cc:729] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc701461b7245e0ba761cec2ab9b8d6, State: Initialized, Role: FOLLOWER
I20260929 18:58:20.676581 1440 consensus_queue.cc:260] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.676842 1440 ts_tablet_manager.cc:1428] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:20.676847 1287 heartbeater.cc:503] Master 127.0.98.62:46227 was elected leader, sending a full tablet report...
I20260929 18:58:20.706123 1446 raft_consensus.cc:482] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:20.706231 1446 raft_consensus.cc:504] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.706580 1446 leader_election.cc:288] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04327123a43544b69122e550fdbf3aec (127.0.98.3:42793), fba39b5b92c14d2dac8f554b95e60725 (127.0.98.1:41989)
I20260929 18:58:20.709595 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.707070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45171 (local address 127.0.98.1:41989)
0929 18:58:20.707138 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.707140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.707147 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:20.707172 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.707174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.707185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.707242 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:20.707444 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.707893 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.708883 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.709242 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.709296 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.709304 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.709481 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.709525 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:58:20.709594 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.706883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41989 (local address 127.0.98.2:45171)
0929 18:58:20.707044 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.707055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.707108 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.707298 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.707299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.707301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:20.707367 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.707369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.708004 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.708005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.708768 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.708770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.709325 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.709327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.709403 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.709407 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.709465 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.709495 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260929 18:58:20.709694 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.706839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56173 (local address 127.0.98.3:42793)
0929 18:58:20.706901 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.706903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.706911 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:20.706974 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.706975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.706987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.707040 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:20.707301 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.707831 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.708877 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.709325 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.709403 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.709412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.709582 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.709624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:58:20.709884 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.706772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42793 (local address 127.0.98.2:56173)
0929 18:58:20.706828 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:20.706840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:20.706898 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:20.707119 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:20.707121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:20.707122 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:20.707199 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.707201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.707969 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.707970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.708761 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:20.708763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.709409 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.709410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:20.709499 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.709502 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:20.709557 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:58:20.709591 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":38}
I20260929 18:58:20.709977 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1152ccb538094600a672419677d03687" candidate_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba39b5b92c14d2dac8f554b95e60725" is_pre_election: true
I20260929 18:58:20.710139 1107 raft_consensus.cc:2454] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc701461b7245e0ba761cec2ab9b8d6 in term 0.
I20260929 18:58:20.710158 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1152ccb538094600a672419677d03687" candidate_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04327123a43544b69122e550fdbf3aec" is_pre_election: true
I20260929 18:58:20.710357 1375 raft_consensus.cc:2454] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc701461b7245e0ba761cec2ab9b8d6 in term 0.
I20260929 18:58:20.710348 1177 leader_election.cc:302] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc701461b7245e0ba761cec2ab9b8d6, fba39b5b92c14d2dac8f554b95e60725; no voters:
I20260929 18:58:20.710494 1446 raft_consensus.cc:2781] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:20.710547 1446 raft_consensus.cc:482] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:20.710578 1446 raft_consensus.cc:3037] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:20.711242 1446 raft_consensus.cc:504] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.711400 1446 leader_election.cc:288] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [CANDIDATE]: Term 1 election: Requested vote from peers 04327123a43544b69122e550fdbf3aec (127.0.98.3:42793), fba39b5b92c14d2dac8f554b95e60725 (127.0.98.1:41989)
I20260929 18:58:20.711585 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1152ccb538094600a672419677d03687" candidate_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba39b5b92c14d2dac8f554b95e60725"
I20260929 18:58:20.711580 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1152ccb538094600a672419677d03687" candidate_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04327123a43544b69122e550fdbf3aec"
I20260929 18:58:20.711647 1375 raft_consensus.cc:3037] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:20.711670 1107 raft_consensus.cc:3037] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:20.712589 1107 raft_consensus.cc:2454] T 1152ccb538094600a672419677d03687 P fba39b5b92c14d2dac8f554b95e60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc701461b7245e0ba761cec2ab9b8d6 in term 1.
I20260929 18:58:20.712598 1375 raft_consensus.cc:2454] T 1152ccb538094600a672419677d03687 P 04327123a43544b69122e550fdbf3aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc701461b7245e0ba761cec2ab9b8d6 in term 1.
I20260929 18:58:20.712779 1177 leader_election.cc:302] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04327123a43544b69122e550fdbf3aec, 2fc701461b7245e0ba761cec2ab9b8d6; no voters:
I20260929 18:58:20.712894 1446 raft_consensus.cc:2781] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:20.713052 1446 raft_consensus.cc:686] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc701461b7245e0ba761cec2ab9b8d6, State: Running, Role: LEADER
I20260929 18:58:20.713155 1446 consensus_queue.cc:237] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } }
I20260929 18:58:20.713994 961 catalog_manager.cc:5495] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc701461b7245e0ba761cec2ab9b8d6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04327123a43544b69122e550fdbf3aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba39b5b92c14d2dac8f554b95e60725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc701461b7245e0ba761cec2ab9b8d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37319 } health_report { overall_health: HEALTHY } } }
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:20.732561 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.725189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.0.98.1:41989)
0929 18:58:20.725267 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.725270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.725390 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:58:20.725469 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.725470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.725483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.725532 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.726137 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.726603 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.727357 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.727553 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.727638 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.729947 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.729956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.729956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.729962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.731542 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.731841 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.731844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.731844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.731903 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.732072 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.732073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.732073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.732206 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.732304 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.732387 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.732449 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:58:20.733631 1288 tablet.cc:2369] T 1152ccb538094600a672419677d03687 P 2fc701461b7245e0ba761cec2ab9b8d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:20.756135 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.748813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.0.98.1:41989)
0929 18:58:20.748873 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.748875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.748976 (+ 101us) server_negotiation.cc:407] Connection header received
0929 18:58:20.749057 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.749058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.749067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.749124 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.749666 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.750281 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.751021 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.751225 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.751314 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.753514 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.753515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.753516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.753523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.754835 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.755236 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.755237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.755237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.755325 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.755543 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.755544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.755544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.755649 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.755771 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.755953 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.756031 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:20.777964 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:20.771245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.0.98.1:41989)
0929 18:58:20.771334 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:20.771336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:20.771423 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:58:20.771475 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:20.771476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:20.771485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:20.771536 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:20.772114 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.772570 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.773314 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:20.773494 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:20.773545 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:20.775773 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:20.775775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:20.775775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:20.775781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:20.777065 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.777361 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.777362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.777362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.777410 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:20.777577 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:20.777578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:20.777578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:20.777662 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:20.777742 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:20.777814 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:58:20.777865 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 18:58:20.778456 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1152ccb538094600a672419677d03687: Uninitialized: clean time has not yet been initialized
I20260929 18:58:20.779337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1024
I20260929 18:58:20.784466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1158
I20260929 18:58:20.788417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1292
I20260929 18:58:20.792743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 930
2026-09-29T18:58:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1955 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[1462](info): set up 2 sockets
Sep 29 18:58:20 dist-test-slave-gfsd krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260929 18:58:21.834846 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:21 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:21.954190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:22.044483 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:22.044721 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:22.044762 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:22.045961 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:22.046031 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:58:22.046058 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:22.046108 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:22.046152 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:22.046186 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:22.047765 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:22.048038 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:22.048281 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:22.053256 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.053277 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:22.053416 1475 server_base.cc:1034] running on GCE node
W20260929 18:58:22.053272 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:22.053840 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:22.054077 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:22.055217 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790708302055194 us; error 37 us; skew 500 ppm
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:22.057679 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:22.058074 1475 webserver.cc:460] Webserver started at http://127.0.98.62:35889/ using document root <none> and password file <none>
I20260929 18:58:22.058249 1475 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:22.058287 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:22.058374 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:22.059448 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "f29860ad12f84e24b0662da62ce36fc8"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "c5b232228874fa7662e713f46a8a1144"
server_key_iv: "45d432451932e667a5df8df8149dc963"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.059751 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "f29860ad12f84e24b0662da62ce36fc8"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "c5b232228874fa7662e713f46a8a1144"
server_key_iv: "45d432451932e667a5df8df8149dc963"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.060954 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:22.061604 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.061770 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:22.061874 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "f29860ad12f84e24b0662da62ce36fc8"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "c5b232228874fa7662e713f46a8a1144"
server_key_iv: "45d432451932e667a5df8df8149dc963"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.061939 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:22.079002 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:22.080948 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:22.081110 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:22.085596 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34287
I20260929 18:58:22.085727 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34287 every 8 connection(s)
I20260929 18:58:22.086007 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:22.086690 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.089023 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Bootstrap starting.
I20260929 18:58:22.089092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1475
I20260929 18:58:22.089174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:22.089330 392 external_mini_cluster.cc:1428] Setting key ef981808a25ed05c48cd39de40a03b6e
I20260929 18:58:22.089725 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.090092 1553 log.cc:824] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:22.090812 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: No bootstrap required, opened a new log
I20260929 18:58:22.093048 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } }
I20260929 18:58:22.093223 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.093254 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29860ad12f84e24b0662da62ce36fc8, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.093374 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } }
I20260929 18:58:22.093444 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:22.093478 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:22.093533 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.094170 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } }
I20260929 18:58:22.094278 1553 leader_election.cc:302] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29860ad12f84e24b0662da62ce36fc8; no voters:
I20260929 18:58:22.094477 1553 leader_election.cc:288] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:22.094502 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:22.094759 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:22.094971 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [term 1 LEADER]: Becoming Leader. State: Replica: f29860ad12f84e24b0662da62ce36fc8, State: Running, Role: LEADER
I20260929 18:58:22.095140 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } }
I20260929 18:58:22.095542 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29860ad12f84e24b0662da62ce36fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } } }
I20260929 18:58:22.095633 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:22.095781 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29860ad12f84e24b0662da62ce36fc8. Latest consensus state: current_term: 1 leader_uuid: "f29860ad12f84e24b0662da62ce36fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29860ad12f84e24b0662da62ce36fc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34287 } } }
I20260929 18:58:22.095840 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8 [sys.catalog]: This master's current role is: LEADER
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260929 18:58:22.096596 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:58:22.096642 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:58:22.096694 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:22.096866 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:22.098346 1567 catalog_manager.cc:1347] Generated new cluster ID: 990cb1479a1b43cd98e57089e3212252
I20260929 18:58:22.098403 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:22.110222 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.089848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.0.98.62:34287)
0929 18:58:22.090026 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.090034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.090121 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:58:22.090594 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.090601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.090646 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.090761 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.091569 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.092503 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.093547 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.094179 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.094304 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.097146 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.097151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.097152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.097164 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.108274 (+ 11110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.108767 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.108770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.108771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.108877 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.109099 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.109100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.109100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.109408 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.109530 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.109687 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.109830 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260929 18:58:22.111922 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:22.112597 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:22.117017 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Generated new TSK 0
I20260929 18:58:22.117169 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:22.142884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34287
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:22.222621 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:22.222842 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:22.222882 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:22.224102 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:22.224165 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:22.224200 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:22.224238 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:22.225672 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:22.225970 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:22.226222 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:22.231182 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.231182 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.231235 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:22.231333 1579 server_base.cc:1034] running on GCE node
I20260929 18:58:22.231660 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:22.231885 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:22.233036 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790708302233025 us; error 41 us; skew 500 ppm
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:22.235484 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:22.235944 1579 webserver.cc:460] Webserver started at http://127.0.98.1:43717/ using document root <none> and password file <none>
I20260929 18:58:22.236166 1579 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:22.236231 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:22.236362 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:22.237426 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb7ddfc7a20043d39b3309d143382ecd"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "0a76b8f05e1d8bc69883ccf3c3db2eae"
server_key_iv: "81afcde9b68350ef93f1fb7f8f639637"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.237789 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb7ddfc7a20043d39b3309d143382ecd"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "0a76b8f05e1d8bc69883ccf3c3db2eae"
server_key_iv: "81afcde9b68350ef93f1fb7f8f639637"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.239084 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:58:22.239902 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.240115 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:22.240190 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "bb7ddfc7a20043d39b3309d143382ecd"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "0a76b8f05e1d8bc69883ccf3c3db2eae"
server_key_iv: "81afcde9b68350ef93f1fb7f8f639637"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.240280 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:22.253515 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:22.255785 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:22.255944 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:22.256434 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:22.256474 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.256495 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:22.256507 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.263476 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37519
I20260929 18:58:22.263569 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37519 every 8 connection(s)
I20260929 18:58:22.263868 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:22.267351 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.256985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49969 (local address 127.0.98.62:34287)
0929 18:58:22.257055 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.257058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.257264 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:58:22.257751 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.257753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.257767 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.257845 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.259203 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.260080 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.261644 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.261991 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.262061 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.264566 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.264568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.264568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.264576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.265973 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.266367 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.266368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.266369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.266425 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.266693 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.266694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.266695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.266811 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.266887 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.267154 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.267238 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:58:22.267519 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.256985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.1:49969)
0929 18:58:22.257138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.257177 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.257658 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.258452 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.258457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.258828 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.259104 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.259111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.260184 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.260185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.261550 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.261553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.262102 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.262103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.262246 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.262802 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.262813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.264468 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.266062 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.266068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.266070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.266289 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.266508 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.266508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.266509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.266623 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.266876 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.266883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.267022 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.267122 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.267270 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260929 18:58:22.267719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1579
I20260929 18:58:22.267812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:22.267992 392 external_mini_cluster.cc:1428] Setting key 205c92da7437a1ecb2a9e6d9e9f10484
I20260929 18:58:22.268011 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34287
I20260929 18:58:22.268078 1710 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:22.268245 1710 heartbeater.cc:511] Master 127.0.98.62:34287 requested a full tablet report, sending...
I20260929 18:58:22.269203 1506 ts_manager.cc:194] Registered new tserver with Master: bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519)
I20260929 18:58:22.269811 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49969
I20260929 18:58:22.275036 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.268961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.1:53493)
0929 18:58:22.269106 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.269116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.269195 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.269352 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.269353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.269566 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.269629 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.269630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.270545 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.270546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.271045 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.271047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.271546 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.271547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.271595 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.271987 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.271991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.272235 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.273990 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.273991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.273991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.274156 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.274380 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.274380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.274381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.274437 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.274767 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.274768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.274821 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.274889 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.274965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260929 18:58:22.275156 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.269011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53493 (local address 127.0.98.62:34287)
0929 18:58:22.269050 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.269052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.269170 (+ 118us) server_negotiation.cc:407] Connection header received
0929 18:58:22.269258 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.269259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.269268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.269294 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.269707 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.270438 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.271128 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.271461 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.271525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.272319 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.272320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.272320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.272327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.273892 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.274236 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.274237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.274237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.274296 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.274506 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.274507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.274507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.274665 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.274743 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.274941 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.275027 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:22.295032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34287
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:22.376750 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:22.376973 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:22.376999 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:22.378201 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:22.378255 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:22.378279 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:22.378300 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:22.379927 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:22.380192 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:22.380446 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:22.385598 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.385613 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.385670 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:22.385807 1719 server_base.cc:1034] running on GCE node
I20260929 18:58:22.387125 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:22.387422 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:22.388597 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790708302388587 us; error 36 us; skew 500 ppm
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:22.391196 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:22.391652 1719 webserver.cc:460] Webserver started at http://127.0.98.2:43967/ using document root <none> and password file <none>
I20260929 18:58:22.391839 1719 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:22.391881 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:22.391968 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:22.392962 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "aa8fb0a714c3ae7bcd4986599a7b16d6"
server_key_iv: "f5c743af28467dbac8e75aa919973eac"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.393261 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "aa8fb0a714c3ae7bcd4986599a7b16d6"
server_key_iv: "f5c743af28467dbac8e75aa919973eac"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.394598 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:22.395520 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.395695 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:22.395776 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "aa8fb0a714c3ae7bcd4986599a7b16d6"
server_key_iv: "f5c743af28467dbac8e75aa919973eac"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.395862 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:22.411274 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:22.413802 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:22.413928 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:22.414425 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:22.414466 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.414491 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:22.414505 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:22.421701 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36179
I20260929 18:58:22.421780 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36179 every 8 connection(s)
I20260929 18:58:22.422053 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:22.425081 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.415063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52357 (local address 127.0.98.62:34287)
0929 18:58:22.415121 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.415124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.415299 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:58:22.415874 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.415875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.415889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.415935 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.416572 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.417374 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.419132 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.419488 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.419563 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.422258 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.422259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.422259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.422268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.423723 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.424148 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.424149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.424150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.424209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.424456 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.424458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.424458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.424574 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.424649 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.424910 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.424985 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:58:22.425285 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.414972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.2:52357)
0929 18:58:22.415138 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.415179 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.415741 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.416028 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.416032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.416278 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.416471 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.416475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.417502 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.417503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.419008 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.419012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.419633 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.419634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.419797 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.420285 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.420296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.422141 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.423823 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.423829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.423831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.424056 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.424274 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.424275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.424275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.424386 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.424641 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.424647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.424783 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.424892 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.425053 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260929 18:58:22.425783 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34287
I20260929 18:58:22.425881 1850 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:22.426059 1850 heartbeater.cc:511] Master 127.0.98.62:34287 requested a full tablet report, sending...
I20260929 18:58:22.426483 1507 ts_manager.cc:194] Registered new tserver with Master: 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.426934 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52357
I20260929 18:58:22.429778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1719
I20260929 18:58:22.429852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:22.430078 392 external_mini_cluster.cc:1428] Setting key 80a59a8d3ee98451e763ac73b0513cfc
I20260929 18:58:22.432298 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.426821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40537 (local address 127.0.98.62:34287)
0929 18:58:22.426861 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.426863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.426991 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:58:22.427065 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.427066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.427076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.427106 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.427525 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.427976 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.428683 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.428881 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.428967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.429729 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.429730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.429730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.429736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.431112 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.431521 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.431522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.431523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.431575 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.431772 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.431774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.431774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.431912 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.431988 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.432143 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.432211 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:58:22.432312 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.426776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.2:40537)
0929 18:58:22.426919 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.426935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.427003 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.427161 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.427161 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.427371 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.427427 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.427428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.428062 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.428063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.428587 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.428588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.428969 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.428969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.429041 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.429384 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.429388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.429636 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.431211 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.431212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.431213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.431440 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.431644 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.431645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.431646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.431701 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.431992 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.431993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.432035 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.432118 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.432212 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:22.454651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34287
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:22.536549 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:22.536789 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:22.536829 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:22.537982 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:22.538048 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:22.538082 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:22.538120 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:22.539680 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:22.539937 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:22.540160 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:22.545325 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.545377 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:22.545393 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:22.545619 1859 server_base.cc:1034] running on GCE node
I20260929 18:58:22.545815 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:22.546065 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:22.547233 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790708302547212 us; error 43 us; skew 500 ppm
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:22.549960 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:22.550446 1859 webserver.cc:460] Webserver started at http://127.0.98.3:36059/ using document root <none> and password file <none>
I20260929 18:58:22.550681 1859 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:22.550727 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:22.550860 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:22.552057 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "09375eda5b4a4d2e915c345108a9ef17"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "43e64c3ed151c496abf35c8dc65ce8eb"
server_key_iv: "378af5ff85ddf51f8807dd9cc023c58d"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.552423 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "09375eda5b4a4d2e915c345108a9ef17"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "43e64c3ed151c496abf35c8dc65ce8eb"
server_key_iv: "378af5ff85ddf51f8807dd9cc023c58d"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.553776 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:58:22.554617 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.554792 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:22.554901 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "09375eda5b4a4d2e915c345108a9ef17"
format_stamp: "Formatted at 2026-09-29 18:58:22 on dist-test-slave-gfsd"
server_key: "43e64c3ed151c496abf35c8dc65ce8eb"
server_key_iv: "378af5ff85ddf51f8807dd9cc023c58d"
server_key_version: "encryptionkey@0"
I20260929 18:58:22.554998 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:22.573276 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:22.575464 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:22.575630 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:22.576224 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:22.576267 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.576318 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:22.576346 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:22.584075 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43419
I20260929 18:58:22.584371 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43419 every 8 connection(s)
I20260929 18:58:22.584504 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:22.586508 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.576699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60065 (local address 127.0.98.62:34287)
0929 18:58:22.576761 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.576763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.576964 (+ 201us) server_negotiation.cc:407] Connection header received
0929 18:58:22.577719 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.577721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.577731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.577786 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.578734 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.579557 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.580680 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.580895 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.580990 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.583709 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.583710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.583710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.583717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.585350 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.585693 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.585693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.585694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.585743 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.585949 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.585950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.585950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.586046 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.586096 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.586388 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.586446 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:58:22.586696 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.576620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.3:60065)
0929 18:58:22.576798 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.576840 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.577587 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.577868 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.577872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.578271 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.578609 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.578616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.579677 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.579678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.580585 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.580587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.581004 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.581005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.581139 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.581666 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.581675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.583603 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.585420 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.585423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.585425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.585627 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.585795 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.585795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.585795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.585892 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.586127 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.586133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.586253 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.586367 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.586491 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260929 18:58:22.587212 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34287
I20260929 18:58:22.587330 1990 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:22.587507 1990 heartbeater.cc:511] Master 127.0.98.62:34287 requested a full tablet report, sending...
I20260929 18:58:22.587913 1506 ts_manager.cc:194] Registered new tserver with Master: 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3:43419)
I20260929 18:58:22.588510 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60065
I20260929 18:58:22.590106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 1859
I20260929 18:58:22.590257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:22.590438 392 external_mini_cluster.cc:1428] Setting key 69cc6614fb7beebc81d976a7ec76c2c1
I20260929 18:58:22.591449 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:22.593968 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.588327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35673 (local address 127.0.98.62:34287)
0929 18:58:22.588367 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.588369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.588438 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:58:22.588493 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.588494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.588502 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.588573 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.588952 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.589414 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.590156 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.590430 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.590500 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.591514 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.591516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.591516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.591523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.592809 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.593212 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.593213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.593214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.593267 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.593471 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.593472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.593473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.593582 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.593658 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.593808 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.593883 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:22.593981 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.588251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.98.3:35673)
0929 18:58:22.588378 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.588388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.588450 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.588588 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.588588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.588778 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.588864 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.588865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.589505 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.589506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.590049 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.590050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.590512 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.590513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.590586 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.591073 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.591079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.591452 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.592959 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.592960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.592960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.593128 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.593337 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.593338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.593339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.593404 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.593649 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.593650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.593693 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.593781 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.593880 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:22.626264 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.617137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.98.62:34287)
0929 18:58:22.618315 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.618317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.618331 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:58:22.618365 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.618367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.618379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.618462 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.619322 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.620252 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.621071 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.621413 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.621482 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.623911 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.623912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.623912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.623919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.625180 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.625498 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.625498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.625499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.625550 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.625769 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.625770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.625770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.625902 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.625976 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.626098 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.626156 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143}
I20260929 18:58:22.628017 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:22.629527 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:22.636314 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.632193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.98.2:36179)
0929 18:58:22.632419 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.632422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.632429 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:22.632478 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.632481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.632499 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.632598 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.632830 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.633942 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635175 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635906 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635973 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.636051 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.636155 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.636196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260929 18:58:22.636330 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.631841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36179 (local address 127.0.0.1:33426)
0929 18:58:22.632305 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.632314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.632350 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.632668 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.632669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.632671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.632744 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.632746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.634040 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.634041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.635097 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.635098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635981 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.636038 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.636051 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.636125 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.636190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":29,"thread_start_us":118,"threads_started":1}
I20260929 18:58:22.636406 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.631850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37519 (local address 127.0.0.1:41114)
0929 18:58:22.632418 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.632426 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.632476 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.632699 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.632699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.632701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.632826 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.632828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.634049 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.634050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.634972 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.634975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635904 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.636026 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.636071 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.636160 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.636205 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":119,"threads_started":1}
I20260929 18:58:22.636330 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.631841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43419 (local address 127.0.0.1:42880)
0929 18:58:22.632082 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.632140 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.632209 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.632653 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.632657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.632659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.632727 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.632729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.633645 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.633646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.634873 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.634876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635950 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.636028 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.636051 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.636126 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.636189 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":22,"thread_start_us":118,"threads_started":1}
I20260929 18:58:22.636690 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.632173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.98.1:37519)
0929 18:58:22.632352 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.632357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.632490 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:58:22.632533 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.632535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.632549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.632600 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.632910 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.633960 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635070 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635793 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635850 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.635918 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.636450 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.636500 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260929 18:58:22.636987 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.632007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.0.98.3:43419)
0929 18:58:22.632422 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.632428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.632440 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:58:22.632485 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.632487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.632503 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.632578 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.632835 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.633544 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.634970 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.635748 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.635825 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.635911 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.636794 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.636850 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":91,"threads_started":1}
I20260929 18:58:22.637233 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 716b300c52444ddea22e1c9f42883b6b (DEFAULT_TABLE table=test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.637292 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 716b300c52444ddea22e1c9f42883b6b (DEFAULT_TABLE table=test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.637615 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b300c52444ddea22e1c9f42883b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.637633 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b300c52444ddea22e1c9f42883b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.639683 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 716b300c52444ddea22e1c9f42883b6b (DEFAULT_TABLE table=test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.640098 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b300c52444ddea22e1c9f42883b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.641855 2014 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Bootstrap starting.
I20260929 18:58:22.641906 2013 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Bootstrap starting.
I20260929 18:58:22.642540 2013 tablet_bootstrap.cc:654] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.642580 2014 tablet_bootstrap.cc:654] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.642896 2014 log.cc:824] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:22.642905 2013 log.cc:824] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:22.643242 2015 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Bootstrap starting.
I20260929 18:58:22.643955 2013 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: No bootstrap required, opened a new log
I20260929 18:58:22.643987 2015 tablet_bootstrap.cc:654] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.644043 2013 ts_tablet_manager.cc:1397] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:22.644186 2014 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: No bootstrap required, opened a new log
I20260929 18:58:22.644258 2014 ts_tablet_manager.cc:1397] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:22.644299 2015 log.cc:824] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:22.645068 2015 tablet_bootstrap.cc:492] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: No bootstrap required, opened a new log
I20260929 18:58:22.645138 2015 ts_tablet_manager.cc:1397] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:22.645762 2013 raft_consensus.cc:348] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.645923 2013 raft_consensus.cc:374] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.645967 2013 raft_consensus.cc:729] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7ddfc7a20043d39b3309d143382ecd, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.646080 2013 consensus_queue.cc:260] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.646266 2014 raft_consensus.cc:348] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.646370 2013 ts_tablet_manager.cc:1428] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:58:22.646427 2014 raft_consensus.cc:374] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.646466 2014 raft_consensus.cc:729] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797d390f2cb04d1b8caaaa5e0bbd188b, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.646497 1710 heartbeater.cc:503] Master 127.0.98.62:34287 was elected leader, sending a full tablet report...
I20260929 18:58:22.646605 2014 consensus_queue.cc:260] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.646860 2014 ts_tablet_manager.cc:1428] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:22.647136 1850 heartbeater.cc:503] Master 127.0.98.62:34287 was elected leader, sending a full tablet report...
I20260929 18:58:22.647076 2015 raft_consensus.cc:348] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.647240 2015 raft_consensus.cc:374] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.647324 2015 raft_consensus.cc:729] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09375eda5b4a4d2e915c345108a9ef17, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.647440 2015 consensus_queue.cc:260] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.647692 2015 ts_tablet_manager.cc:1428] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 18:58:22.647727 1990 heartbeater.cc:503] Master 127.0.98.62:34287 was elected leader, sending a full tablet report...
W20260929 18:58:22.672601 1851 tablet.cc:2369] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:22.691506 2019 raft_consensus.cc:482] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:22.691665 2019 raft_consensus.cc:504] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.692081 2019 leader_election.cc:288] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3:43419), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.695178 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.692423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36289 (local address 127.0.98.2:36179)
0929 18:58:22.692503 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.692505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.692622 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:58:22.692669 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.692697 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.692712 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.692788 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.692974 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.693433 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694430 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.694810 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.694904 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.695083 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.695119 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:58:22.695195 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.692273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36179 (local address 127.0.98.1:36289)
0929 18:58:22.692531 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.692544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.692614 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.692807 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.692808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.692810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.692888 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.692890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.693517 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.693518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.694345 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.694348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.694895 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.694984 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.694990 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.695059 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.695102 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260929 18:58:22.695303 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.692282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43419 (local address 127.0.98.1:43517)
0929 18:58:22.692562 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.692571 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.692614 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.692792 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.692793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.692795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.692868 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.692870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.693495 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.693496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.694335 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.694338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.694990 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.695066 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.695068 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.695130 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.695179 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":20,"thread_start_us":130,"threads_started":1}
I20260929 18:58:22.695303 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.692417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43517 (local address 127.0.98.3:43419)
0929 18:58:22.692463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.692465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.692671 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:58:22.692699 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.692700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.692712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.692787 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.692948 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.693415 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694443 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.694912 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.694991 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.695000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.695162 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.695199 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:58:22.695641 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b300c52444ddea22e1c9f42883b6b" candidate_uuid: "bb7ddfc7a20043d39b3309d143382ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" is_pre_election: true
I20260929 18:58:22.695694 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b300c52444ddea22e1c9f42883b6b" candidate_uuid: "bb7ddfc7a20043d39b3309d143382ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09375eda5b4a4d2e915c345108a9ef17" is_pre_election: true
I20260929 18:58:22.695823 1804 raft_consensus.cc:2454] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb7ddfc7a20043d39b3309d143382ecd in term 0.
I20260929 18:58:22.695848 1944 raft_consensus.cc:2454] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb7ddfc7a20043d39b3309d143382ecd in term 0.
I20260929 18:58:22.696081 1596 leader_election.cc:302] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 797d390f2cb04d1b8caaaa5e0bbd188b, bb7ddfc7a20043d39b3309d143382ecd; no voters:
I20260929 18:58:22.696233 2019 raft_consensus.cc:2781] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:22.696316 2019 raft_consensus.cc:482] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:22.696357 2019 raft_consensus.cc:3037] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.697732 2019 raft_consensus.cc:504] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.697885 2019 leader_election.cc:288] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [CANDIDATE]: Term 1 election: Requested vote from peers 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3:43419), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.698035 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b300c52444ddea22e1c9f42883b6b" candidate_uuid: "bb7ddfc7a20043d39b3309d143382ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09375eda5b4a4d2e915c345108a9ef17"
I20260929 18:58:22.698066 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b300c52444ddea22e1c9f42883b6b" candidate_uuid: "bb7ddfc7a20043d39b3309d143382ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
I20260929 18:58:22.698120 1944 raft_consensus.cc:3037] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.698145 1804 raft_consensus.cc:3037] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.698997 1944 raft_consensus.cc:2454] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7ddfc7a20043d39b3309d143382ecd in term 1.
I20260929 18:58:22.698997 1804 raft_consensus.cc:2454] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7ddfc7a20043d39b3309d143382ecd in term 1.
I20260929 18:58:22.699193 1596 leader_election.cc:302] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 797d390f2cb04d1b8caaaa5e0bbd188b, bb7ddfc7a20043d39b3309d143382ecd; no voters:
I20260929 18:58:22.699328 2019 raft_consensus.cc:2781] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:22.699515 2019 raft_consensus.cc:686] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 1 LEADER]: Becoming Leader. State: Replica: bb7ddfc7a20043d39b3309d143382ecd, State: Running, Role: LEADER
I20260929 18:58:22.699625 2019 consensus_queue.cc:237] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260929 18:58:22.700549 1506 catalog_manager.cc:5495] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bb7ddfc7a20043d39b3309d143382ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } health_report { overall_health: HEALTHY } } }
I20260929 18:58:22.716204 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.713608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.1:37519)
0929 18:58:22.713675 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.713677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.713688 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:22.713791 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.713792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.713817 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.713891 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:22.714046 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.714510 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.715336 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.715535 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.715622 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.715850 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.715962 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.716047 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.716099 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.716120 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:58:22.717926 1944 raft_consensus.cc:1261] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Refusing update from remote peer bb7ddfc7a20043d39b3309d143382ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.717943 1804 raft_consensus.cc:1261] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Refusing update from remote peer bb7ddfc7a20043d39b3309d143382ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.718255 2019 consensus_queue.cc:1035] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.718431 2023 consensus_queue.cc:1035] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.720543 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334741207930089472 to 7334741207858401280. Current Snapshot: MvccSnapshot[applied={T|T < 7334741207930089472}]
I20260929 18:58:22.723946 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:22.724054 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:22.725021 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Sending DeleteTablet for 3 replicas of tablet 716b300c52444ddea22e1c9f42883b6b
I20260929 18:58:22.725309 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b300c52444ddea22e1c9f42883b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42880
I20260929 18:58:22.725376 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b300c52444ddea22e1c9f42883b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33426
I20260929 18:58:22.725559 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b300c52444ddea22e1c9f42883b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 18:58:22.725612 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:22.725751 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:22.725778 2038 tablet_replica.cc:330] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: stopping tablet replica
I20260929 18:58:22.725869 2038 raft_consensus.cc:2229] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:22.726095 2038 raft_consensus.cc:2258] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.726109 2036 tablet_replica.cc:330] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: stopping tablet replica
I20260929 18:58:22.726382 2036 raft_consensus.cc:2229] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:22.726514 2038 ts_tablet_manager.cc:1905] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.726540 2036 raft_consensus.cc:2258] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.727082 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 (DEFAULT_TABLE table=test-table [id=eff043d80c26441fbe01882081972d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.727121 2036 ts_tablet_manager.cc:1905] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.727201 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e71071cc0545fbb46f974f7dc9ee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.727485 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 (DEFAULT_TABLE table=test-table [id=eff043d80c26441fbe01882081972d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.727579 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e71071cc0545fbb46f974f7dc9ee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.728543 2038 ts_tablet_manager.cc:1918] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:22.728605 2038 log.cc:1196] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/716b300c52444ddea22e1c9f42883b6b
I20260929 18:58:22.728855 2015 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Bootstrap starting.
I20260929 18:58:22.728969 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 (DEFAULT_TABLE table=test-table [id=eff043d80c26441fbe01882081972d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:22.729059 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e71071cc0545fbb46f974f7dc9ee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.729257 2015 tablet_bootstrap.cc:654] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.729324 2014 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Bootstrap starting.
I20260929 18:58:22.729602 2038 ts_tablet_manager.cc:1939] T 716b300c52444ddea22e1c9f42883b6b P bb7ddfc7a20043d39b3309d143382ecd: Deleting consensus metadata
I20260929 18:58:22.729748 2014 tablet_bootstrap.cc:654] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.730034 1491 catalog_manager.cc:4841] TS bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519): tablet 716b300c52444ddea22e1c9f42883b6b (table test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]) successfully deleted
I20260929 18:58:22.730280 2015 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: No bootstrap required, opened a new log
I20260929 18:58:22.730324 2015 ts_tablet_manager.cc:1397] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:22.730417 2014 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: No bootstrap required, opened a new log
I20260929 18:58:22.730465 2014 ts_tablet_manager.cc:1397] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:22.730471 2015 raft_consensus.cc:348] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.730553 2015 raft_consensus.cc:374] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.730576 2015 raft_consensus.cc:729] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09375eda5b4a4d2e915c345108a9ef17, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.730616 2015 consensus_queue.cc:260] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.730607 2014 raft_consensus.cc:348] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.730669 2014 raft_consensus.cc:374] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.730690 2014 raft_consensus.cc:729] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797d390f2cb04d1b8caaaa5e0bbd188b, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.730702 2015 ts_tablet_manager.cc:1428] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.730731 2014 consensus_queue.cc:260] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.730777 2013 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Bootstrap starting.
I20260929 18:58:22.730811 2014 ts_tablet_manager.cc:1428] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.731621 2013 tablet_bootstrap.cc:654] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.732564 2013 tablet_bootstrap.cc:492] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: No bootstrap required, opened a new log
I20260929 18:58:22.732625 2013 ts_tablet_manager.cc:1397] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:58:22.732748 2013 raft_consensus.cc:348] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.732816 2013 raft_consensus.cc:374] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.732836 2013 raft_consensus.cc:729] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7ddfc7a20043d39b3309d143382ecd, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.732872 2013 consensus_queue.cc:260] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.732956 2013 ts_tablet_manager.cc:1428] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.733170 2036 ts_tablet_manager.cc:1918] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:22.733238 2036 log.cc:1196] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/716b300c52444ddea22e1c9f42883b6b
I20260929 18:58:22.733498 2036 ts_tablet_manager.cc:1939] T 716b300c52444ddea22e1c9f42883b6b P 09375eda5b4a4d2e915c345108a9ef17: Deleting consensus metadata
I20260929 18:58:22.733914 2037 tablet_replica.cc:330] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: stopping tablet replica
I20260929 18:58:22.733922 1493 catalog_manager.cc:4841] TS 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3:43419): tablet 716b300c52444ddea22e1c9f42883b6b (table test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]) successfully deleted
I20260929 18:58:22.734020 2037 raft_consensus.cc:2229] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:22.734128 2037 raft_consensus.cc:2258] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.734517 2037 ts_tablet_manager.cc:1905] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.735778 2037 ts_tablet_manager.cc:1918] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:22.735836 2037 log.cc:1196] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/716b300c52444ddea22e1c9f42883b6b
I20260929 18:58:22.736093 2037 ts_tablet_manager.cc:1939] T 716b300c52444ddea22e1c9f42883b6b P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting consensus metadata
I20260929 18:58:22.736481 1492 catalog_manager.cc:4841] TS 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179): tablet 716b300c52444ddea22e1c9f42883b6b (table test-table [id=cbcc6d8c4b6d4116a4834cc685f19399]) successfully deleted
W20260929 18:58:22.764621 1711 tablet.cc:2369] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:22.805414 2021 raft_consensus.cc:482] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:22.805522 2021 raft_consensus.cc:504] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.805857 2021 leader_election.cc:288] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.809382 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.806228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41761 (local address 127.0.98.1:37519)
0929 18:58:22.806275 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.806278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.806345 (+ 67us) server_negotiation.cc:407] Connection header received
0929 18:58:22.806404 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.806405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.806413 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.806473 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.806659 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.807355 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.808634 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.809012 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.809097 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.809107 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.809265 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.809307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:58:22.809412 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.806031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36179 (local address 127.0.98.3:45229)
0929 18:58:22.806085 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.806098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.806142 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.806326 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.806327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.806328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.806400 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.806402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.807184 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.807185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.808501 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.808504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.809199 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.809199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.809266 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.809268 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.809315 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.809350 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:58:22.809648 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.806133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45229 (local address 127.0.98.2:36179)
0929 18:58:22.806177 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.806180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.806188 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:22.806214 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.806215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.806224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.806273 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:22.806457 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.807083 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.808595 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.809116 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.809195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.809207 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.809516 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.809567 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:58:22.809419 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.806038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37519 (local address 127.0.98.3:41761)
0929 18:58:22.806289 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.806298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.806349 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.806486 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.806487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.806488 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:22.806565 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.806566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.807438 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.807438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.808534 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.808536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.809124 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.809125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.809201 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.809205 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.809259 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.809309 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260929 18:58:22.809840 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e71071cc0545fbb46f974f7dc9ee85" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" is_pre_election: true
I20260929 18:58:22.809935 1804 raft_consensus.cc:2454] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 0.
I20260929 18:58:22.810142 1876 leader_election.cc:302] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09375eda5b4a4d2e915c345108a9ef17, 797d390f2cb04d1b8caaaa5e0bbd188b; no voters:
I20260929 18:58:22.810268 2021 raft_consensus.cc:2781] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:22.810308 2021 raft_consensus.cc:482] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:22.810338 2021 raft_consensus.cc:3037] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.810278 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e71071cc0545fbb46f974f7dc9ee85" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7ddfc7a20043d39b3309d143382ecd" is_pre_election: true
I20260929 18:58:22.810384 1664 raft_consensus.cc:2454] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 0.
I20260929 18:58:22.811090 2021 raft_consensus.cc:504] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.811237 2021 leader_election.cc:288] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 election: Requested vote from peers bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.811383 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e71071cc0545fbb46f974f7dc9ee85" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
I20260929 18:58:22.811388 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e71071cc0545fbb46f974f7dc9ee85" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7ddfc7a20043d39b3309d143382ecd"
I20260929 18:58:22.811455 1664 raft_consensus.cc:3037] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.811455 1804 raft_consensus.cc:3037] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.812060 1804 raft_consensus.cc:2454] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 1.
I20260929 18:58:22.812217 1876 leader_election.cc:302] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09375eda5b4a4d2e915c345108a9ef17, 797d390f2cb04d1b8caaaa5e0bbd188b; no voters:
I20260929 18:58:22.812239 1664 raft_consensus.cc:2454] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 1.
I20260929 18:58:22.812320 2021 raft_consensus.cc:2781] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:22.812472 2021 raft_consensus.cc:686] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 1 LEADER]: Becoming Leader. State: Replica: 09375eda5b4a4d2e915c345108a9ef17, State: Running, Role: LEADER
I20260929 18:58:22.812564 2021 consensus_queue.cc:237] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } }
I20260929 18:58:22.813337 1506 catalog_manager.cc:5495] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } health_report { overall_health: HEALTHY } } }
W20260929 18:58:22.835083 1991 tablet.cc:2369] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:22.854825 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.848037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34287 (local address 127.0.0.1:37656)
0929 18:58:22.848198 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.848217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.848278 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.848450 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.848451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.848782 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:22.848830 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.848832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.849493 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.849493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.850011 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.850012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.850384 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.850384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.850430 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.850985 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:22.850991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:22.852441 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:22.853821 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.853827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.853829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.854078 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.854258 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:22.854259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:22.854259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:22.854316 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:22.854522 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:22.854528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:22.854569 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.854632 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.854727 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260929 18:58:22.854869 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.848098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.98.62:34287)
0929 18:58:22.848164 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.848166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.848279 (+ 113us) server_negotiation.cc:407] Connection header received
0929 18:58:22.848325 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.848326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.848335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.848429 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.848914 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.849401 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.850120 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.850309 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.850372 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.852550 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:22.852551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:22.852551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:22.852558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:22.853751 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.854152 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.854152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.854153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.854200 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:22.854369 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:22.854369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:22.854370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:22.854470 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:22.854519 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.854666 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.854772 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":41}
I20260929 18:58:22.857072 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37656:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:58:22.857267 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:22.858858 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7e25c359dc364dfeba05e2287722570c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8713aca36145445abe4712316bee2b1a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:22.858938 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 7e25c359dc364dfeba05e2287722570c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8713aca36145445abe4712316bee2b1a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:22.859042 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e25c359dc364dfeba05e2287722570c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.859055 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e25c359dc364dfeba05e2287722570c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.859432 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7e25c359dc364dfeba05e2287722570c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8713aca36145445abe4712316bee2b1a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:22.859591 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e25c359dc364dfeba05e2287722570c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:22.860576 2013 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: Bootstrap starting.
I20260929 18:58:22.861270 2015 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: Bootstrap starting.
I20260929 18:58:22.861366 2013 tablet_bootstrap.cc:654] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.861655 2015 tablet_bootstrap.cc:654] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.861668 2014 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: Bootstrap starting.
I20260929 18:58:22.861974 2013 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: No bootstrap required, opened a new log
I20260929 18:58:22.862020 2013 ts_tablet_manager.cc:1397] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:22.862066 2014 tablet_bootstrap.cc:654] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:22.862104 2013 raft_consensus.cc:348] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.862195 2013 raft_consensus.cc:374] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.862219 2013 raft_consensus.cc:729] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7ddfc7a20043d39b3309d143382ecd, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.862283 2015 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: No bootstrap required, opened a new log
I20260929 18:58:22.862272 2013 consensus_queue.cc:260] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.862321 2015 ts_tablet_manager.cc:1397] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:22.862371 2013 ts_tablet_manager.cc:1428] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.862403 2031 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.862506 2031 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.862473 2015 raft_consensus.cc:348] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.862551 2015 raft_consensus.cc:374] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.862586 2015 raft_consensus.cc:729] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09375eda5b4a4d2e915c345108a9ef17, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.862633 2015 consensus_queue.cc:260] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.862733 2015 ts_tablet_manager.cc:1428] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.862866 2021 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.862957 2021 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.863253 2014 tablet_bootstrap.cc:492] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: No bootstrap required, opened a new log
I20260929 18:58:22.863329 2014 ts_tablet_manager.cc:1397] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:22.863477 2014 raft_consensus.cc:348] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.863569 2014 raft_consensus.cc:374] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:22.863590 2014 raft_consensus.cc:729] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797d390f2cb04d1b8caaaa5e0bbd188b, State: Initialized, Role: FOLLOWER
I20260929 18:58:22.863642 2014 consensus_queue.cc:260] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.863732 2014 ts_tablet_manager.cc:1428] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:22.863798 2020 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.863909 2020 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.937801 2021 raft_consensus.cc:482] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:22.937892 2021 raft_consensus.cc:504] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.938071 2021 leader_election.cc:288] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.938334 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e25c359dc364dfeba05e2287722570c" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7ddfc7a20043d39b3309d143382ecd" is_pre_election: true
I20260929 18:58:22.938362 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e25c359dc364dfeba05e2287722570c" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" is_pre_election: true
I20260929 18:58:22.938448 1804 raft_consensus.cc:2454] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 0.
I20260929 18:58:22.938454 1664 raft_consensus.cc:2454] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 0.
I20260929 18:58:22.938624 1876 leader_election.cc:302] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09375eda5b4a4d2e915c345108a9ef17, 797d390f2cb04d1b8caaaa5e0bbd188b; no voters:
I20260929 18:58:22.938743 2021 raft_consensus.cc:2781] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:22.938793 2021 raft_consensus.cc:482] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:22.938823 2021 raft_consensus.cc:3037] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.939769 2021 raft_consensus.cc:504] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.939911 2021 leader_election.cc:288] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 election: Requested vote from peers bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519), 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179)
I20260929 18:58:22.940049 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e25c359dc364dfeba05e2287722570c" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b"
I20260929 18:58:22.940073 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e25c359dc364dfeba05e2287722570c" candidate_uuid: "09375eda5b4a4d2e915c345108a9ef17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7ddfc7a20043d39b3309d143382ecd"
I20260929 18:58:22.940156 1804 raft_consensus.cc:3037] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.940155 1664 raft_consensus.cc:3037] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:22.940717 1804 raft_consensus.cc:2454] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 1.
I20260929 18:58:22.940780 1664 raft_consensus.cc:2454] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09375eda5b4a4d2e915c345108a9ef17 in term 1.
I20260929 18:58:22.940970 1875 leader_election.cc:302] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09375eda5b4a4d2e915c345108a9ef17, bb7ddfc7a20043d39b3309d143382ecd; no voters:
I20260929 18:58:22.941075 2021 raft_consensus.cc:2781] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:22.941141 2021 raft_consensus.cc:686] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [term 1 LEADER]: Becoming Leader. State: Replica: 09375eda5b4a4d2e915c345108a9ef17, State: Running, Role: LEADER
I20260929 18:58:22.941186 2021 consensus_queue.cc:237] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } }
I20260929 18:58:22.941535 2043 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: TxnStatusTablet state changed. Reason: New leader 09375eda5b4a4d2e915c345108a9ef17. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.941623 2043 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:22.941814 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:58:22.941937 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:58:22.941859 1507 catalog_manager.cc:5495] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:22.958264 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.955234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43419 (local address 127.0.0.1:42884)
0929 18:58:22.955359 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.955374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.955437 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.955635 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.955636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.955638 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:22.955688 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.955690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.956630 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.956631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.957219 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.957221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.957709 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.957710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.957765 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.957787 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.958089 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.958100 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.958152 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.958204 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 18:58:22.958261 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.955367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.98.3:43419)
0929 18:58:22.955434 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.955436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.955448 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:58:22.955505 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.955506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.955526 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.955601 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:22.955766 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.956540 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.957305 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.957627 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.957697 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.957876 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.958002 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.958084 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.958156 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.958177 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:58:22.959559 1804 raft_consensus.cc:1261] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Refusing update from remote peer 09375eda5b4a4d2e915c345108a9ef17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.959623 1664 raft_consensus.cc:1261] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Refusing update from remote peer 09375eda5b4a4d2e915c345108a9ef17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.959987 2021 consensus_queue.cc:1035] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.960153 2043 consensus_queue.cc:1035] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.960785 2020 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: TxnStatusTablet state changed. Reason: New leader 09375eda5b4a4d2e915c345108a9ef17. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.960880 2020 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.961126 2031 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: TxnStatusTablet state changed. Reason: New leader 09375eda5b4a4d2e915c345108a9ef17. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.961220 2031 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.961661 2043 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.961741 2043 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:22.961807 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334741208920047616 to 7334741208847544320. Current Snapshot: MvccSnapshot[applied={T|T < 7334741208920047616 or (T in {7334741208920047616})}]
I20260929 18:58:22.962265 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334741208920047616 to 7334741208847544320. Current Snapshot: MvccSnapshot[applied={T|T < 7334741208920047616}]
I20260929 18:58:22.962381 2020 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.962450 2020 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 797d390f2cb04d1b8caaaa5e0bbd188b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.963126 2021 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.963204 2021 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P 09375eda5b4a4d2e915c345108a9ef17: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:22.963641 2031 tablet_replica.cc:439] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09375eda5b4a4d2e915c345108a9ef17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } } peers { permanent_uuid: "09375eda5b4a4d2e915c345108a9ef17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43419 } } peers { permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 } } }
I20260929 18:58:22.963730 2031 tablet_replica.cc:442] T 7e25c359dc364dfeba05e2287722570c P bb7ddfc7a20043d39b3309d143382ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:22.966441 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.963868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.0.98.3:43419)
0929 18:58:22.963899 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.963901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.963907 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:22.963926 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.963926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.963935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.963978 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:22.964209 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.964734 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.965776 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.965975 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.966105 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.966240 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.966265 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.966300 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.966383 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.966397 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:58:22.971179 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.969002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46879 (local address 127.0.98.3:43419)
0929 18:58:22.969048 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.969050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.969059 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:22.969078 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.969079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.969085 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.969164 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:22.969270 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.969708 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.970515 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.970687 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.970766 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.970931 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.970963 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.971028 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:22.971087 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:58:22.971100 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:58:22.971184 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.968808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43419 (local address 127.0.98.3:46879)
0929 18:58:22.968899 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:22.968910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:22.968952 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:22.969168 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:22.969168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:22.969169 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:22.969219 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.969220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.969788 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.969789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.970420 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:22.970422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.970766 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.970767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:22.970823 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:22.970836 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:22.971028 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:22.971030 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:22.971083 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:58:22.971119 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:58:22.973338 1664 raft_consensus.cc:1261] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Refusing update from remote peer 09375eda5b4a4d2e915c345108a9ef17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.973371 1804 raft_consensus.cc:1261] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Refusing update from remote peer 09375eda5b4a4d2e915c345108a9ef17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:22.973591 2062 consensus_queue.cc:1035] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797d390f2cb04d1b8caaaa5e0bbd188b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.973675 2043 consensus_queue.cc:1035] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7ddfc7a20043d39b3309d143382ecd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:22.985092 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:22.985208 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:22.985824 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f29860ad12f84e24b0662da62ce36fc8: Sending DeleteTablet for 3 replicas of tablet a7e71071cc0545fbb46f974f7dc9ee85
I20260929 18:58:22.986068 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33426
I20260929 18:58:22.986196 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 18:58:22.986277 2038 tablet_replica.cc:330] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: stopping tablet replica
I20260929 18:58:22.986330 2038 raft_consensus.cc:2229] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:22.986369 2038 raft_consensus.cc:2258] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.986603 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e71071cc0545fbb46f974f7dc9ee85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42880
I20260929 18:58:22.986745 2038 ts_tablet_manager.cc:1905] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.988641 2038 ts_tablet_manager.cc:1918] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:22.988713 2038 log.cc:1196] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/a7e71071cc0545fbb46f974f7dc9ee85
I20260929 18:58:22.989024 2038 ts_tablet_manager.cc:1939] T a7e71071cc0545fbb46f974f7dc9ee85 P bb7ddfc7a20043d39b3309d143382ecd: Deleting consensus metadata
I20260929 18:58:22.989331 1491 catalog_manager.cc:4841] TS bb7ddfc7a20043d39b3309d143382ecd (127.0.98.1:37519): tablet a7e71071cc0545fbb46f974f7dc9ee85 (table test-table [id=eff043d80c26441fbe01882081972d42]) successfully deleted
I20260929 18:58:22.990474 2037 tablet_replica.cc:330] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: stopping tablet replica
I20260929 18:58:22.990558 2037 raft_consensus.cc:2229] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:22.990602 2037 raft_consensus.cc:2258] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.990679 2036 tablet_replica.cc:330] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: stopping tablet replica
I20260929 18:58:22.990739 2036 raft_consensus.cc:2229] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:22.990825 2036 raft_consensus.cc:2258] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:22.991142 2037 ts_tablet_manager.cc:1905] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.992010 2036 ts_tablet_manager.cc:1905] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:22.992662 2037 ts_tablet_manager.cc:1918] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:22.992715 2037 log.cc:1196] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/a7e71071cc0545fbb46f974f7dc9ee85
I20260929 18:58:22.992944 2037 ts_tablet_manager.cc:1939] T a7e71071cc0545fbb46f974f7dc9ee85 P 797d390f2cb04d1b8caaaa5e0bbd188b: Deleting consensus metadata
I20260929 18:58:22.993310 1492 catalog_manager.cc:4841] TS 797d390f2cb04d1b8caaaa5e0bbd188b (127.0.98.2:36179): tablet a7e71071cc0545fbb46f974f7dc9ee85 (table test-table [id=eff043d80c26441fbe01882081972d42]) successfully deleted
I20260929 18:58:22.993552 2036 ts_tablet_manager.cc:1918] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:22.993614 2036 log.cc:1196] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/a7e71071cc0545fbb46f974f7dc9ee85
I20260929 18:58:22.993820 2036 ts_tablet_manager.cc:1939] T a7e71071cc0545fbb46f974f7dc9ee85 P 09375eda5b4a4d2e915c345108a9ef17: Deleting consensus metadata
I20260929 18:58:22.994244 1493 catalog_manager.cc:4841] TS 09375eda5b4a4d2e915c345108a9ef17 (127.0.98.3:43419): tablet a7e71071cc0545fbb46f974f7dc9ee85 (table test-table [id=eff043d80c26441fbe01882081972d42]) successfully deleted
Sep 29 18:58:22 dist-test-slave-gfsd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:23.004894 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:22.992825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.1:37519)
0929 18:58:22.994625 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:22.994627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:22.994637 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:22.994673 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:22.994673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:22.994684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:22.994748 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:22.995851 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.996632 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.997753 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:22.998130 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:22.998206 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:23.001958 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:23.001970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:23.001971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:23.001979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:23.003698 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:23.004073 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:23.004077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:23.004077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:23.004137 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:23.004337 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:23.004338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:23.004338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:23.004501 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:23.004667 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:23.004707 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 18:58:23.004789 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775}
W20260929 18:58:23.005149 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41138
I20260929 18:58:23.012365 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:23.006122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.62:34287)
0929 18:58:23.006172 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:23.006174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:23.006184 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:23.006247 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:23.006248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:23.006258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:23.006313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:23.006897 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:23.007484 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:23.008293 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:23.008503 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:23.008585 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:23.009755 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:23.009757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:23.009757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:23.009764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:23.011207 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:23.011598 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:23.011599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:23.011600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:23.011658 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:23.011863 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:23.011864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:23.011864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:23.011979 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:23.012077 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:23.012179 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:58:23.012259 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 18:58:23.012636 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37664
I20260929 18:58:23.013295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1579
I20260929 18:58:23.018975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1719
I20260929 18:58:23.023913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1859
I20260929 18:58:23.029042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 1475
2026-09-29T18:58:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2235 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:23 dist-test-slave-gfsd krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:23 dist-test-slave-gfsd krb5kdc[2072](info): set up 2 sockets
Sep 29 18:58:23 dist-test-slave-gfsd krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260929 18:58:24.067022 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:24.186137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:24.265271 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:24.265504 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:24.265543 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:24.266662 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:24.266726 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:58:24.266748 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:24.266772 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:24.266799 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:24.266827 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:24.268251 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:24.268513 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:24.268765 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:24.273900 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.273906 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.274042 2085 server_base.cc:1034] running on GCE node
W20260929 18:58:24.274240 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.274500 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:24.274798 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:24.275981 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790708304275962 us; error 35 us; skew 500 ppm
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:24.278620 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:24.279084 2085 webserver.cc:460] Webserver started at http://127.0.98.62:37327/ using document root <none> and password file <none>
I20260929 18:58:24.279356 2085 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:24.279402 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:24.279556 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:24.280613 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "7fdd544775c245b4aaebd684121aca92"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "95ba51dae41cea3a6d1ab89834892335"
server_key_iv: "e36cab84a68765733bdf61638e98e1fc"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.280978 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fdd544775c245b4aaebd684121aca92"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "95ba51dae41cea3a6d1ab89834892335"
server_key_iv: "e36cab84a68765733bdf61638e98e1fc"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.282312 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:24.283099 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.283324 2085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:24.283442 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "7fdd544775c245b4aaebd684121aca92"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "95ba51dae41cea3a6d1ab89834892335"
server_key_iv: "e36cab84a68765733bdf61638e98e1fc"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.283535 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:24.316381 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:24.318570 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:24.318746 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:24.323137 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39815
I20260929 18:58:24.323211 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39815 every 8 connection(s)
I20260929 18:58:24.323572 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:24.324265 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:24.326639 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Bootstrap starting.
I20260929 18:58:24.327258 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:24.327574 2163 log.cc:824] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:24.328281 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: No bootstrap required, opened a new log
I20260929 18:58:24.330507 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 18:58:24.330689 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:24.330739 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fdd544775c245b4aaebd684121aca92, State: Initialized, Role: FOLLOWER
I20260929 18:58:24.330853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2085
I20260929 18:58:24.330945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:24.330937 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 18:58:24.331027 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:24.331072 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:24.331133 392 external_mini_cluster.cc:1428] Setting key bf907bf0ce36c010473092b21ea3091f
I20260929 18:58:24.331125 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:24.331910 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 18:58:24.332039 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fdd544775c245b4aaebd684121aca92; no voters:
I20260929 18:58:24.332206 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:24.332259 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:24.332407 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [term 1 LEADER]: Becoming Leader. State: Replica: 7fdd544775c245b4aaebd684121aca92, State: Running, Role: LEADER
I20260929 18:58:24.332476 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:24.332504 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 18:58:24.332799 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fdd544775c245b4aaebd684121aca92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } } }
I20260929 18:58:24.332893 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:24.332800 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fdd544775c245b4aaebd684121aca92. Latest consensus state: current_term: 1 leader_uuid: "7fdd544775c245b4aaebd684121aca92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd544775c245b4aaebd684121aca92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } } }
I20260929 18:58:24.333109 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:24.333218 2173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:24.333964 2173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:24.335610 2173 catalog_manager.cc:1347] Generated new cluster ID: 3c1494a34db1450aa02f55594e9da141
I20260929 18:58:24.335661 2173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:24.344221 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.331679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.0.98.62:39815)
0929 18:58:24.331854 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.331862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.331888 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:58:24.331989 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.331995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.332062 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.332249 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.333537 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.334959 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.336052 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.336665 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.336767 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.339649 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.339654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.339654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.339666 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.342227 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.342641 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.342645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.342645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.342757 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.342961 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.342962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.342962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.343259 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.343446 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.343866 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.344002 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260929 18:58:24.356613 2173 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:24.357141 2173 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:24.363449 2173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Generated new TSK 0
I20260929 18:58:24.363667 2173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:24.398484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:24.478499 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:24.478689 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:24.478718 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:24.479797 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:24.479833 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:24.479853 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:58:24.479864 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:24.479888 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:24.481204 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:24.481379 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:24.481544 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:58:24.486720 2189 server_base.cc:1034] running on GCE node
W20260929 18:58:24.486683 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.486676 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.486662 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.487068 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:24.487255 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:24.488457 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790708304488446 us; error 29 us; skew 500 ppm
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:24.490980 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:24.491477 2189 webserver.cc:460] Webserver started at http://127.0.98.1:37583/ using document root <none> and password file <none>
I20260929 18:58:24.491695 2189 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:24.491740 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:24.491899 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:24.492957 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "1012737b25e64cf1925d967744e151b9"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a2ab1c263bd799f2097403894aaa94ec"
server_key_iv: "25e4c45a366a60b73977d71e3a90d6d4"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.493319 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1012737b25e64cf1925d967744e151b9"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a2ab1c263bd799f2097403894aaa94ec"
server_key_iv: "25e4c45a366a60b73977d71e3a90d6d4"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.494674 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:24.495558 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.495776 2189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:24.495877 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "1012737b25e64cf1925d967744e151b9"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a2ab1c263bd799f2097403894aaa94ec"
server_key_iv: "25e4c45a366a60b73977d71e3a90d6d4"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.495954 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:24.506531 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:24.508602 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:24.508766 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:24.509265 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:24.509327 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.509392 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:24.509420 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.516511 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43063
I20260929 18:58:24.516633 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43063 every 8 connection(s)
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:24.516947 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:24.520283 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.509787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33691 (local address 127.0.98.62:39815)
0929 18:58:24.509829 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.509831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.510132 (+ 301us) server_negotiation.cc:407] Connection header received
0929 18:58:24.510696 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.510698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.510716 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.510798 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.511675 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.512545 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.514115 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.514471 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.514548 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.517343 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.517345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.517345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.517352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.518897 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.519331 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.519332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.519332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.519394 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.519649 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.519650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.519650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.519763 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.519839 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.520109 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.520197 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:24.520453 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.509779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.1:33691)
0929 18:58:24.509988 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.510035 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.510554 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.510883 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.510893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.511257 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.511558 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.511565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.512685 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.512686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.514036 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.514039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.514565 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.514566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.514720 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.515268 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.515297 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.517253 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.519005 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.519011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.519013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.519231 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.519463 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.519464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.519464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.519576 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.519830 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.519836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.519972 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.520070 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.520221 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":105,"threads_started":1}
I20260929 18:58:24.521013 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 18:58:24.521092 2320 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:24.521267 2320 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
I20260929 18:58:24.521936 2117 ts_manager.cc:194] Registered new tserver with Master: 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063)
I20260929 18:58:24.522876 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33691
I20260929 18:58:24.523116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2189
I20260929 18:58:24.523214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:24.523404 392 external_mini_cluster.cc:1428] Setting key 8881360c11fdb3d8235e29a36080bec6
I20260929 18:58:24.528353 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.522006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.1:60191)
0929 18:58:24.522169 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.522180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.522256 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.522396 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.522397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.522599 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.522663 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.522665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.523578 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.523579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.524211 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.524213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.524623 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.524624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.524705 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.525179 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.525183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.525445 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.527262 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.527263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.527263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.527541 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.527788 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.527789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.527789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.527842 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.528130 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.528131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.528165 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.528211 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.528284 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260929 18:58:24.528475 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.522105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60191 (local address 127.0.98.62:39815)
0929 18:58:24.522142 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.522144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.522278 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:58:24.522305 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.522306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.522315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.522368 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.522733 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.523491 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.524324 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.524531 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.524623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.525539 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.525541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.525541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.525549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.527180 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.527633 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.527635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.527635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.527704 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.527905 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.527906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.527907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.528053 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.528130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.528241 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.528341 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:24.550405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:24.633168 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:24.633457 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:24.633486 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:24.634755 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:24.634799 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:24.634822 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:58:24.634882 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:24.634918 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:24.636420 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:24.636659 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:24.636919 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:24.642148 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.642161 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.642180 2329 server_base.cc:1034] running on GCE node
W20260929 18:58:24.642167 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.642592 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:24.642841 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:24.644029 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790708304644014 us; error 47 us; skew 500 ppm
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:24.646516 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:24.646938 2329 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260929 18:58:24.647158 2329 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:24.647203 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:24.647372 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:24.648430 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "cedc8a46e33244feb0c2d464161b432d"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a729dc5185417bd14bc9e4cbf276317a"
server_key_iv: "47c2a991665d9c2c4810fd42904ec5d3"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.648823 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cedc8a46e33244feb0c2d464161b432d"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a729dc5185417bd14bc9e4cbf276317a"
server_key_iv: "47c2a991665d9c2c4810fd42904ec5d3"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.650264 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:24.651026 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.651221 2329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:24.651340 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "cedc8a46e33244feb0c2d464161b432d"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "a729dc5185417bd14bc9e4cbf276317a"
server_key_iv: "47c2a991665d9c2c4810fd42904ec5d3"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.651422 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:24.665480 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:24.667734 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:24.667901 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:24.668401 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:24.668462 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.668507 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:24.668551 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:24.675827 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33829
I20260929 18:58:24.675933 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33829 every 8 connection(s)
I20260929 18:58:24.676188 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:24.680086 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.668975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41833 (local address 127.0.98.62:39815)
0929 18:58:24.669049 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.669052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.669303 (+ 251us) server_negotiation.cc:407] Connection header received
0929 18:58:24.670048 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.670049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.670088 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.670177 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.671118 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.671756 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.673423 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.673652 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.673747 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.676161 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.676163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.676163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.676172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.678169 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.679049 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.679050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.679051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.679117 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.679430 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.679431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.679431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.679552 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.679631 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.679916 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.679995 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:58:24.680279 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.668931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.2:41833)
0929 18:58:24.669146 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.669195 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.669899 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.670283 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.670289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.670684 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.670995 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.671001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.671833 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.671834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.673312 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.673315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.673758 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.673760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.673900 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.674452 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.674461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.676047 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.678299 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.678306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.678308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.678820 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.679192 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.679193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.679193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.679359 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.679622 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.679628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.679768 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.679891 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.680049 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260929 18:58:24.680806 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 18:58:24.680908 2460 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:24.681090 2460 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
I20260929 18:58:24.681506 2116 ts_manager.cc:194] Registered new tserver with Master: cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829)
I20260929 18:58:24.681996 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41833
I20260929 18:58:24.685388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2329
I20260929 18:58:24.685556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:24.685765 392 external_mini_cluster.cc:1428] Setting key 8d03f67baf6b51fb61e3cee1d85c1b50
I20260929 18:58:24.687795 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.681803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44069 (local address 127.0.98.62:39815)
0929 18:58:24.681840 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.681842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.681969 (+ 127us) server_negotiation.cc:407] Connection header received
0929 18:58:24.682050 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.682051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.682061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.682108 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.682481 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.683114 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.683853 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.684082 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.684137 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.684986 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.684987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.684987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.684993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.686543 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.686967 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.686968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.686968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.687028 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.687239 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.687239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.687239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.687454 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.687519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.687676 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.687737 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 18:58:24.687912 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.681751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.2:44069)
0929 18:58:24.681891 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.681901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.681993 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.682147 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.682148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.682337 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.682406 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.682408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.683206 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.683207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.683782 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.683785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.684150 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.684151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.684203 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.684636 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.684641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.684909 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.686656 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.686657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.686658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.686870 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.687104 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.687105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.687105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.687174 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.687528 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.687529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.687575 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.687656 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.687765 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:24.714309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:24.794037 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:24.794277 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:24.794318 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:24.795517 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:24.795585 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:24.795624 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:58:24.795670 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:24.795710 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:24.797171 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:24.797435 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:24.797672 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:24.802706 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.802706 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:24.802696 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:24.802954 2469 server_base.cc:1034] running on GCE node
I20260929 18:58:24.803148 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:24.803431 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:24.804585 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790708304804576 us; error 25 us; skew 500 ppm
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:24.807029 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:24.807448 2469 webserver.cc:460] Webserver started at http://127.0.98.3:38239/ using document root <none> and password file <none>
I20260929 18:58:24.807643 2469 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:24.807683 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:24.807770 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:24.808727 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "d50b35bf33861d7142e853aaf19933dd"
server_key_iv: "d4b1ca145e2fba810fe52c090a862703"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.809031 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "d50b35bf33861d7142e853aaf19933dd"
server_key_iv: "d4b1ca145e2fba810fe52c090a862703"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.810230 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:24.810945 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.811136 2469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:24.811209 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
format_stamp: "Formatted at 2026-09-29 18:58:24 on dist-test-slave-gfsd"
server_key: "d50b35bf33861d7142e853aaf19933dd"
server_key_iv: "d4b1ca145e2fba810fe52c090a862703"
server_key_version: "encryptionkey@0"
I20260929 18:58:24.811259 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:24.833442 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:24.836309 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:24.836478 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:24.837126 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:24.837170 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:24.837195 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:24.837234 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:24.845118 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36065
I20260929 18:58:24.845191 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36065 every 8 connection(s)
I20260929 18:58:24.845569 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:24.847810 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.837624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33255 (local address 127.0.98.62:39815)
0929 18:58:24.837698 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.837701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.837865 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:58:24.838401 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.838403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.838419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.838511 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.839361 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.840157 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.841714 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.842046 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.842133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.844825 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.844827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.844827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.844837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.846506 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.846868 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.846869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.846870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.846925 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.847191 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.847192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.847192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.847380 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.847451 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.847677 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.847745 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:58:24.847947 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.837583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.3:33255)
0929 18:58:24.837737 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.837776 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.838281 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.838570 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.838573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.838928 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.839209 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.839214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.840277 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.840278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.841614 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.841617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.842154 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.842154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.842286 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.842779 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.842790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.844709 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.846583 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.846588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.846590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.846802 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.847023 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.847023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.847024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.847119 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.847450 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.847454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.847571 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.847658 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.847776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260929 18:58:24.848469 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 18:58:24.848548 2600 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:24.848732 2600 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
I20260929 18:58:24.849128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2469
I20260929 18:58:24.849241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:24.849447 392 external_mini_cluster.cc:1428] Setting key ff211f9519ac375b68c27980dbb319f7
I20260929 18:58:24.849632 2117 ts_manager.cc:194] Registered new tserver with Master: 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:24.850001 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33255
I20260929 18:58:24.850194 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:24.856781 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.851012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33001 (local address 127.0.98.62:39815)
0929 18:58:24.851054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.851056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.851114 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:58:24.851186 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.851186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.851196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.851250 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.851607 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.852074 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.852760 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.852980 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.853064 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.854076 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.854077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.854078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.854084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.855659 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.856012 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.856013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.856013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.856065 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.856286 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.856287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.856287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.856398 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.856472 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.856618 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.856689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:58:24.856794 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.850941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.3:33001)
0929 18:58:24.851060 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.851071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.851129 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.851256 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.851256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.851471 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:24.851529 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.851530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.852164 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.852166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.852678 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.852680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.853066 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.853067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.853143 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.853582 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:24.853593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:24.854000 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:24.855755 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.855755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.855756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.855927 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.856155 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:24.856156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:24.856157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:24.856212 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:24.856463 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:24.856464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:24.856506 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.856593 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.856691 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:58:24 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:24.873224 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.865978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.0.98.62:39815)
0929 18:58:24.866038 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.866039 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.866194 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:58:24.866218 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.866219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.866234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.866292 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:24.866876 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.867440 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.868210 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.868409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.868483 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.870726 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:24.870727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:24.870728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:24.870735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:24.872231 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.872535 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.872536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.872536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.872585 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:24.872746 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:24.872746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:24.872746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:24.872836 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:24.872972 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.873085 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.873143 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:58:24.874902 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:24.876549 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:24.883302 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.878798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33829 (local address 127.0.0.1:41190)
0929 18:58:24.878923 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.878967 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.879055 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.879594 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.879598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.879600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:24.879669 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.879671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.880989 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.880990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.882106 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.882109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.882844 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.883043 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.883048 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.883111 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.883184 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":4,"thread_start_us":134,"threads_started":1}
I20260929 18:58:24.883302 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.879047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.0.1:46158)
0929 18:58:24.879683 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.879693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.879744 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.879983 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.879984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.879986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:24.880027 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.880028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.881293 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.881295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.882257 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.882260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.882936 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.883021 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.883035 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.883123 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.883186 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":25,"thread_start_us":66,"threads_started":1}
I20260929 18:58:24.883579 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.879011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.0.98.2:33829)
0929 18:58:24.879397 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.879401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.879411 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:24.879459 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.879461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.879475 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.879586 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:24.879770 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.880900 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882195 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.882755 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882819 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.882863 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.883436 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.883484 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":74,"threads_started":1}
I20260929 18:58:24.883349 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.879165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.0.98.3:36065)
0929 18:58:24.879366 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.879371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.879813 (+ 442us) server_negotiation.cc:407] Connection header received
0929 18:58:24.879872 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.879874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.879890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.879978 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:24.880111 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.881182 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882345 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.882846 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.882981 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.883178 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.883225 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260929 18:58:24.884241 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9608376ee4a83b9ae52f457f76349 (DEFAULT_TABLE table=test-table [id=d2fad0b3ade6450faf8509ec916dbe00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:24.884521 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d9608376ee4a83b9ae52f457f76349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:24.884441 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9608376ee4a83b9ae52f457f76349 (DEFAULT_TABLE table=test-table [id=d2fad0b3ade6450faf8509ec916dbe00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:24.884701 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d9608376ee4a83b9ae52f457f76349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:24.887420 2623 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Bootstrap starting.
I20260929 18:58:24.887905 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.879210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.0.98.1:43063)
0929 18:58:24.879369 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.879384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.879451 (+ 67us) server_negotiation.cc:407] Connection header received
0929 18:58:24.879537 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.879539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.879554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.879624 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:24.879820 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.880686 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.882921 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.883723 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.883791 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.887688 (+ 3897us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:24.887727 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:58:24.887782 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260929 18:58:24.888140 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.878815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:48740)
0929 18:58:24.879397 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.879406 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.879465 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.879691 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.879692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.879694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:24.879738 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.879740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.880761 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.880762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.882108 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.882110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.883929 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.883930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.884009 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.884014 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:24.884066 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:58:24.888050 (+ 3984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260929 18:58:24.888374 2624 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Bootstrap starting.
I20260929 18:58:24.888434 2623 tablet_bootstrap.cc:654] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:24.888816 2623 log.cc:824] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:24.888942 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9608376ee4a83b9ae52f457f76349 (DEFAULT_TABLE table=test-table [id=d2fad0b3ade6450faf8509ec916dbe00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:24.889307 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d9608376ee4a83b9ae52f457f76349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:24.889613 2623 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: No bootstrap required, opened a new log
I20260929 18:58:24.889693 2623 ts_tablet_manager.cc:1397] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:24.891783 2624 tablet_bootstrap.cc:654] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:24.892107 2624 log.cc:824] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:24.892109 2623 raft_consensus.cc:348] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.892333 2623 raft_consensus.cc:374] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:24.892390 2623 raft_consensus.cc:729] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7cbda3010b4fc2bce7b6ec2def0dd5, State: Initialized, Role: FOLLOWER
I20260929 18:58:24.892520 2623 consensus_queue.cc:260] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.892808 2623 ts_tablet_manager.cc:1428] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:24.892833 2600 heartbeater.cc:503] Master 127.0.98.62:39815 was elected leader, sending a full tablet report...
I20260929 18:58:24.893004 2624 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: No bootstrap required, opened a new log
I20260929 18:58:24.893052 2628 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Bootstrap starting.
I20260929 18:58:24.893105 2624 ts_tablet_manager.cc:1397] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:58:24.893678 2628 tablet_bootstrap.cc:654] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:24.894058 2628 log.cc:824] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:24.894837 2628 tablet_bootstrap.cc:492] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: No bootstrap required, opened a new log
I20260929 18:58:24.894918 2628 ts_tablet_manager.cc:1397] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:24.895077 2624 raft_consensus.cc:348] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.895195 2624 raft_consensus.cc:374] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:24.895215 2624 raft_consensus.cc:729] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Initialized, Role: FOLLOWER
I20260929 18:58:24.895351 2624 consensus_queue.cc:260] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.895609 2624 ts_tablet_manager.cc:1428] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:58:24.895681 2460 heartbeater.cc:503] Master 127.0.98.62:39815 was elected leader, sending a full tablet report...
I20260929 18:58:24.896612 2628 raft_consensus.cc:348] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.896760 2628 raft_consensus.cc:374] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:24.896807 2628 raft_consensus.cc:729] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Initialized, Role: FOLLOWER
I20260929 18:58:24.896932 2628 consensus_queue.cc:260] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.897176 2628 ts_tablet_manager.cc:1428] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:58:24.897235 2320 heartbeater.cc:503] Master 127.0.98.62:39815 was elected leader, sending a full tablet report...
W20260929 18:58:24.927060 2461 tablet.cc:2369] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:24.995884 2630 raft_consensus.cc:482] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:24.996045 2630 raft_consensus.cc:504] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:24.996416 2630 leader_election.cc:288] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.000209 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.996674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57103 (local address 127.0.98.1:43063)
0929 18:58:24.996719 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.996721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.996936 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:58:24.997005 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.997006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.997018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.997084 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:24.997302 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.998038 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.999477 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.999888 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.999958 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:24.999968 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.000133 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.000159 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:58:25.000244 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.996620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.98.2:57103)
0929 18:58:24.996871 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.996883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.996947 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.997138 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.997139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.997141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:24.997224 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.997226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.998132 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.998133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.999393 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.999396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.999980 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.999981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.000057 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.000061 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.000127 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.000172 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":67}
I20260929 18:58:25.000628 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.996800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59133 (local address 127.0.98.3:36065)
0929 18:58:24.996837 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:24.996839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:24.996945 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:58:24.997005 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:24.997006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:24.997018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:24.997084 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:24.997309 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.998047 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.999649 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.000097 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.000159 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.000169 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.000514 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.000561 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:58:25.000566 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d9608376ee4a83b9ae52f457f76349" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9" is_pre_election: true
I20260929 18:58:25.000653 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:24.996731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.98.2:59133)
0929 18:58:24.996871 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:24.996883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:24.996947 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:24.997138 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:24.997139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:24.997141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:24.997225 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.997226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:24.998133 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:24.998134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:24.999563 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:24.999565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.000355 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.000356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.000446 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.000450 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.000507 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.000545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":37,"thread_start_us":94,"threads_started":1}
I20260929 18:58:25.000741 2274 raft_consensus.cc:2454] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:25.001008 2345 leader_election.cc:302] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:25.001124 2630 raft_consensus.cc:2781] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:25.001181 2630 raft_consensus.cc:482] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:25.001205 2630 raft_consensus.cc:3037] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.001279 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d9608376ee4a83b9ae52f457f76349" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" is_pre_election: true
I20260929 18:58:25.001384 2554 raft_consensus.cc:2454] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:25.002262 2630 raft_consensus.cc:504] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.002389 2630 leader_election.cc:288] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Requested vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.002524 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d9608376ee4a83b9ae52f457f76349" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9"
I20260929 18:58:25.002547 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d9608376ee4a83b9ae52f457f76349" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
I20260929 18:58:25.002600 2274 raft_consensus.cc:3037] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.002614 2554 raft_consensus.cc:3037] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.003266 2274 raft_consensus.cc:2454] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:25.003412 2554 raft_consensus.cc:2454] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:25.003499 2345 leader_election.cc:302] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:25.003582 2630 raft_consensus.cc:2781] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:25.003749 2630 raft_consensus.cc:686] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 1 LEADER]: Becoming Leader. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Running, Role: LEADER
I20260929 18:58:25.003870 2630 consensus_queue.cc:237] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.004765 2117 catalog_manager.cc:5495] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d reported cstate change: term changed from 0 to 1, leader changed from <none> to cedc8a46e33244feb0c2d464161b432d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
W20260929 18:58:25.017709 2321 tablet.cc:2369] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:58:25 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:25.047159 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.040195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.0.1:55916)
0929 18:58:25.040339 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.040357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.040429 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.040624 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.040625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.040972 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:25.041021 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.041023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.041697 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.041698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.042229 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.042231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.042582 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.042583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.042629 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.043095 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:25.043100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:25.044549 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:25.046157 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:25.046162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:25.046164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:25.046341 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:25.046552 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:25.046552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:25.046552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:25.046605 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:25.046849 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:25.046879 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:25.046930 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.046999 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.047078 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260929 18:58:25.047230 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.040289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.0.98.62:39815)
0929 18:58:25.040391 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.040393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.040406 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:58:25.040517 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.040518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.040527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.040585 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:25.041113 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.041614 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.042323 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.042510 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.042567 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.044627 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:25.044628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:25.044629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:25.044634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:25.046087 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:25.046426 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:25.046427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:25.046427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:25.046483 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:25.046678 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:25.046679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:25.046679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:25.046776 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:25.046833 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.047072 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.047148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:58:25.049346 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55916:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:58:25.049548 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:25.051057 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 064d926ea6454cef98b96fa1dc846a1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315242ced873447dab12b1135b5b2c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:25.051083 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 064d926ea6454cef98b96fa1dc846a1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315242ced873447dab12b1135b5b2c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:25.051086 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 064d926ea6454cef98b96fa1dc846a1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315242ced873447dab12b1135b5b2c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:25.051241 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064d926ea6454cef98b96fa1dc846a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.051241 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064d926ea6454cef98b96fa1dc846a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.051242 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064d926ea6454cef98b96fa1dc846a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.052556 2628 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: Bootstrap starting.
I20260929 18:58:25.053001 2628 tablet_bootstrap.cc:654] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.053013 2623 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Bootstrap starting.
I20260929 18:58:25.053611 2623 tablet_bootstrap.cc:654] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.053728 2628 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: No bootstrap required, opened a new log
I20260929 18:58:25.053799 2628 ts_tablet_manager.cc:1397] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:25.053979 2628 raft_consensus.cc:348] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.054075 2628 raft_consensus.cc:374] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.054066 2624 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: Bootstrap starting.
I20260929 18:58:25.054101 2628 raft_consensus.cc:729] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.054157 2628 consensus_queue.cc:260] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.054296 2628 ts_tablet_manager.cc:1428] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:25.054497 2624 tablet_bootstrap.cc:654] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.054471 2631 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.054557 2631 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.055042 2623 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: No bootstrap required, opened a new log
I20260929 18:58:25.055099 2623 ts_tablet_manager.cc:1397] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:25.055229 2624 tablet_bootstrap.cc:492] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: No bootstrap required, opened a new log
I20260929 18:58:25.055330 2624 ts_tablet_manager.cc:1397] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:25.055327 2623 raft_consensus.cc:348] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.055420 2623 raft_consensus.cc:374] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.055455 2623 raft_consensus.cc:729] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7cbda3010b4fc2bce7b6ec2def0dd5, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.055510 2623 consensus_queue.cc:260] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.055490 2624 raft_consensus.cc:348] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.055567 2624 raft_consensus.cc:374] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.055600 2624 raft_consensus.cc:729] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.055613 2623 ts_tablet_manager.cc:1428] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:25.055652 2624 consensus_queue.cc:260] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.055748 2624 ts_tablet_manager.cc:1428] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:25.055757 2627 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.055838 2627 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.055969 2630 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.056387 2630 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 18:58:25.096891 2601 tablet.cc:2369] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:25.137979 2630 raft_consensus.cc:482] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:25.138073 2630 raft_consensus.cc:504] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.138192 2630 leader_election.cc:288] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.138453 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064d926ea6454cef98b96fa1dc846a1e" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9" is_pre_election: true
I20260929 18:58:25.138545 2274 raft_consensus.cc:2454] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:25.138495 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064d926ea6454cef98b96fa1dc846a1e" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" is_pre_election: true
I20260929 18:58:25.138609 2554 raft_consensus.cc:2454] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:25.138698 2345 leader_election.cc:302] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:25.138829 2630 raft_consensus.cc:2781] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:25.138934 2630 raft_consensus.cc:482] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:25.138957 2630 raft_consensus.cc:3037] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.139688 2630 raft_consensus.cc:504] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.139817 2630 leader_election.cc:288] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Requested vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.140064 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064d926ea6454cef98b96fa1dc846a1e" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9"
I20260929 18:58:25.140142 2274 raft_consensus.cc:3037] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.140120 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064d926ea6454cef98b96fa1dc846a1e" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
I20260929 18:58:25.140220 2554 raft_consensus.cc:3037] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.140763 2274 raft_consensus.cc:2454] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:25.140928 2345 leader_election.cc:302] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:25.140996 2554 raft_consensus.cc:2454] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:25.141078 2630 raft_consensus.cc:2781] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:25.141135 2630 raft_consensus.cc:686] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [term 1 LEADER]: Becoming Leader. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Running, Role: LEADER
I20260929 18:58:25.141179 2630 consensus_queue.cc:237] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.141530 2633 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: TxnStatusTablet state changed. Reason: New leader cedc8a46e33244feb0c2d464161b432d. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.141629 2633 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:25.141882 2645 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:58:25.141858 2117 catalog_manager.cc:5495] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d reported cstate change: term changed from 0 to 1, leader changed from <none> to cedc8a46e33244feb0c2d464161b432d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:25.142007 2645 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:58:25.152644 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.150157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.98.2:33829)
0929 18:58:25.150221 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.150224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.150233 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:25.150265 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.150266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.150291 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.150349 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.150501 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.150958 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.151798 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.152009 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.152092 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.152292 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.152393 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.152492 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.152518 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.152540 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:58:25.152626 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.149996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33829 (local address 127.0.0.1:41206)
0929 18:58:25.150128 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.150139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.150185 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.150379 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.150380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.150382 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:25.150434 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.150436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.151047 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.151048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.151699 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.151701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.152093 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.152093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.152159 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.152180 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.152459 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.152469 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.152505 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.152557 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 18:58:25.154597 2554 raft_consensus.cc:1261] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.154853 2274 raft_consensus.cc:1261] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.154990 2630 consensus_queue.cc:1035] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.155170 2630 consensus_queue.cc:1035] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.156210 2627 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: TxnStatusTablet state changed. Reason: New leader cedc8a46e33244feb0c2d464161b432d. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.156324 2627 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.156296 2631 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: TxnStatusTablet state changed. Reason: New leader cedc8a46e33244feb0c2d464161b432d. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.156386 2631 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.157637 2633 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.157727 2633 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:25.158092 2630 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.158164 2630 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P cedc8a46e33244feb0c2d464161b432d: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:25.158353 2631 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.158418 2631 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 1012737b25e64cf1925d967744e151b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.158799 2648 mvcc.cc:205] Tried to move back new op lower bound from 7334741217910497280 to 7334741217858609152. Current Snapshot: MvccSnapshot[applied={T|T < 7334741217910497280 or (T in {7334741217910497280})}]
I20260929 18:58:25.158938 2650 mvcc.cc:205] Tried to move back new op lower bound from 7334741217910497280 to 7334741217858609152. Current Snapshot: MvccSnapshot[applied={T|T < 7334741217910497280}]
I20260929 18:58:25.159143 2627 tablet_replica.cc:439] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } }
I20260929 18:58:25.159243 2627 tablet_replica.cc:442] T 064d926ea6454cef98b96fa1dc846a1e P 2a7cbda3010b4fc2bce7b6ec2def0dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:25.160087 2649 mvcc.cc:205] Tried to move back new op lower bound from 7334741217910497280 to 7334741217858609152. Current Snapshot: MvccSnapshot[applied={T|T < 7334741217910497280}]
I20260929 18:58:25.162424 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.159753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.0.98.2:33829)
0929 18:58:25.159899 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.159900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.159906 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:25.159922 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.159923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.159930 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.159972 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.160148 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.160708 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.161810 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.162014 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.162069 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.162229 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.162253 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.162290 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.162341 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.162356 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 18:58:25.167375 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.164974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43987 (local address 127.0.98.2:33829)
0929 18:58:25.165006 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.165007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.165055 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:58:25.165125 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.165126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.165134 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.165203 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.165320 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.165837 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.166665 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.166848 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.166938 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.167112 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.167143 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.167207 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.167265 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.167301 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:25.167392 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.164929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33829 (local address 127.0.98.2:43987)
0929 18:58:25.165010 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.165020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.165069 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.165203 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.165204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.165205 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:25.165258 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.165259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.165918 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.165918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.166562 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.166564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.166927 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.166929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.166991 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.167012 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.167206 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.167207 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.167261 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.167319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260929 18:58:25.169381 2554 raft_consensus.cc:1261] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.169395 2274 raft_consensus.cc:1261] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.169580 2659 consensus_queue.cc:1035] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.169683 2633 consensus_queue.cc:1035] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.182098 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:25.182214 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:25.182972 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Sending DeleteTablet for 3 replicas of tablet 08d9608376ee4a83b9ae52f457f76349
I20260929 18:58:25.183476 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9608376ee4a83b9ae52f457f76349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260929 18:58:25.183526 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9608376ee4a83b9ae52f457f76349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41190
I20260929 18:58:25.183548 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9608376ee4a83b9ae52f457f76349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48740
I20260929 18:58:25.183915 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:58:25.183986 2665 tablet_replica.cc:330] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: stopping tablet replica
W20260929 18:58:25.184073 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:25.184094 2665 raft_consensus.cc:2229] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:25.184156 2665 raft_consensus.cc:2258] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:25.184324 2664 tablet_replica.cc:330] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: stopping tablet replica
I20260929 18:58:25.184631 2663 tablet_replica.cc:330] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: stopping tablet replica
I20260929 18:58:25.184738 2663 raft_consensus.cc:2229] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:25.184784 2663 raft_consensus.cc:2258] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:25.184903 2664 raft_consensus.cc:2229] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:25.184998 2664 raft_consensus.cc:2258] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:25.185254 2254 tablet_service.cc:1467] Processing CreateTablet for tablet bb17b8c4618c44e2a3585e3c530e279f (DEFAULT_TABLE table=test-table [id=6d5f0984c2234e568031f58225dd1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:25.185286 2534 tablet_service.cc:1467] Processing CreateTablet for tablet bb17b8c4618c44e2a3585e3c530e279f (DEFAULT_TABLE table=test-table [id=6d5f0984c2234e568031f58225dd1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:25.185367 2390 tablet_service.cc:1467] Processing CreateTablet for tablet bb17b8c4618c44e2a3585e3c530e279f (DEFAULT_TABLE table=test-table [id=6d5f0984c2234e568031f58225dd1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:25.185416 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb17b8c4618c44e2a3585e3c530e279f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.185420 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb17b8c4618c44e2a3585e3c530e279f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.185456 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb17b8c4618c44e2a3585e3c530e279f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:25.186755 2623 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Bootstrap starting.
I20260929 18:58:25.186781 2624 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Bootstrap starting.
I20260929 18:58:25.186795 2628 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Bootstrap starting.
I20260929 18:58:25.187175 2623 tablet_bootstrap.cc:654] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.187175 2624 tablet_bootstrap.cc:654] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.187184 2628 tablet_bootstrap.cc:654] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:25.187790 2663 ts_tablet_manager.cc:1905] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:25.188553 2623 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: No bootstrap required, opened a new log
I20260929 18:58:25.188618 2623 ts_tablet_manager.cc:1397] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:25.188773 2623 raft_consensus.cc:348] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.188848 2623 raft_consensus.cc:374] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.188882 2623 raft_consensus.cc:729] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7cbda3010b4fc2bce7b6ec2def0dd5, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.188920 2623 consensus_queue.cc:260] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.188997 2623 ts_tablet_manager.cc:1428] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:25.189646 2664 ts_tablet_manager.cc:1905] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:25.191107 2664 ts_tablet_manager.cc:1918] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:25.191180 2664 log.cc:1196] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/08d9608376ee4a83b9ae52f457f76349
I20260929 18:58:25.191401 2663 ts_tablet_manager.cc:1918] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:25.191453 2664 ts_tablet_manager.cc:1939] T 08d9608376ee4a83b9ae52f457f76349 P cedc8a46e33244feb0c2d464161b432d: Deleting consensus metadata
I20260929 18:58:25.191483 2663 log.cc:1196] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/08d9608376ee4a83b9ae52f457f76349
I20260929 18:58:25.191499 2665 ts_tablet_manager.cc:1905] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:25.191735 2663 ts_tablet_manager.cc:1939] T 08d9608376ee4a83b9ae52f457f76349 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting consensus metadata
I20260929 18:58:25.191965 2102 catalog_manager.cc:4841] TS cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829): tablet 08d9608376ee4a83b9ae52f457f76349 (table test-table [id=d2fad0b3ade6450faf8509ec916dbe00]) successfully deleted
I20260929 18:58:25.192162 2101 catalog_manager.cc:4841] TS 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065): tablet 08d9608376ee4a83b9ae52f457f76349 (table test-table [id=d2fad0b3ade6450faf8509ec916dbe00]) successfully deleted
I20260929 18:58:25.192447 2628 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: No bootstrap required, opened a new log
I20260929 18:58:25.192507 2628 ts_tablet_manager.cc:1397] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 18:58:25.192656 2628 raft_consensus.cc:348] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.192739 2628 raft_consensus.cc:374] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.192759 2628 raft_consensus.cc:729] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.192799 2628 consensus_queue.cc:260] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.192886 2628 ts_tablet_manager.cc:1428] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:25.193163 2665 ts_tablet_manager.cc:1918] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:25.193236 2665 log.cc:1196] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/08d9608376ee4a83b9ae52f457f76349
I20260929 18:58:25.193511 2665 ts_tablet_manager.cc:1939] T 08d9608376ee4a83b9ae52f457f76349 P 1012737b25e64cf1925d967744e151b9: Deleting consensus metadata
I20260929 18:58:25.193881 2101 catalog_manager.cc:4841] TS 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063): tablet 08d9608376ee4a83b9ae52f457f76349 (table test-table [id=d2fad0b3ade6450faf8509ec916dbe00]) successfully deleted
I20260929 18:58:25.194147 2624 tablet_bootstrap.cc:492] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: No bootstrap required, opened a new log
I20260929 18:58:25.194187 2624 ts_tablet_manager.cc:1397] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 18:58:25.194329 2624 raft_consensus.cc:348] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.194386 2624 raft_consensus.cc:374] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:25.194401 2624 raft_consensus.cc:729] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Initialized, Role: FOLLOWER
I20260929 18:58:25.194453 2624 consensus_queue.cc:260] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.194567 2624 ts_tablet_manager.cc:1428] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:25.273375 2631 raft_consensus.cc:482] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:25.273486 2631 raft_consensus.cc:504] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.273836 2631 leader_election.cc:288] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.277231 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.274124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40923 (local address 127.0.98.3:36065)
0929 18:58:25.274173 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.274175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.274261 (+ 86us) server_negotiation.cc:407] Connection header received
0929 18:58:25.274397 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.274398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.274409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.274466 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:25.274680 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.275538 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.276497 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.276852 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.276949 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.276960 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.277130 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.277169 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:58:25.277244 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.274025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.98.1:40923)
0929 18:58:25.274187 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.274200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.274291 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.274504 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.274505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.274507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:25.274585 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.274592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.275599 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.275600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.276384 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.276386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.276952 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.276953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.277048 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.277051 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.277112 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.277154 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260929 18:58:25.277462 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb17b8c4618c44e2a3585e3c530e279f" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" is_pre_election: true
I20260929 18:58:25.277540 2554 raft_consensus.cc:2454] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 0.
I20260929 18:58:25.277715 2205 leader_election.cc:302] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, 2a7cbda3010b4fc2bce7b6ec2def0dd5; no voters:
I20260929 18:58:25.277850 2631 raft_consensus.cc:2781] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:25.277912 2631 raft_consensus.cc:482] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:25.277942 2631 raft_consensus.cc:3037] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.278241 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.274146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52347 (local address 127.0.98.2:33829)
0929 18:58:25.274194 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.274197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.274282 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:58:25.274395 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.274396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.274406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.274489 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:25.274730 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.275473 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.276959 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.277489 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.277537 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.277549 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.278140 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.278183 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:25.278258 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.274025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33829 (local address 127.0.98.1:52347)
0929 18:58:25.274225 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.274233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.274305 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.274532 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.274533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.274536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:25.274633 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.274635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.275554 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.275555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.276845 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.276848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.277985 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.277986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.278064 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.278067 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.278122 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.278162 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260929 18:58:25.278517 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb17b8c4618c44e2a3585e3c530e279f" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc8a46e33244feb0c2d464161b432d" is_pre_election: true
I20260929 18:58:25.278671 2414 raft_consensus.cc:2454] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 0.
I20260929 18:58:25.278972 2631 raft_consensus.cc:504] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.279137 2631 leader_election.cc:288] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 election: Requested vote from peers cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:25.279266 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb17b8c4618c44e2a3585e3c530e279f" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc8a46e33244feb0c2d464161b432d"
I20260929 18:58:25.279323 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb17b8c4618c44e2a3585e3c530e279f" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
I20260929 18:58:25.279373 2414 raft_consensus.cc:3037] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.279402 2554 raft_consensus.cc:3037] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:25.279950 2554 raft_consensus.cc:2454] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 1.
I20260929 18:58:25.280138 2205 leader_election.cc:302] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, 2a7cbda3010b4fc2bce7b6ec2def0dd5; no voters:
I20260929 18:58:25.280231 2631 raft_consensus.cc:2781] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:25.280313 2414 raft_consensus.cc:2454] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 1.
I20260929 18:58:25.280371 2631 raft_consensus.cc:686] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Running, Role: LEADER
I20260929 18:58:25.280440 2631 consensus_queue.cc:237] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:25.281239 2116 catalog_manager.cc:5495] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1012737b25e64cf1925d967744e151b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1012737b25e64cf1925d967744e151b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:25.309566 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.306686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.0.98.1:43063)
0929 18:58:25.306821 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.306823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.306830 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:25.306866 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.306867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.306889 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.306963 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.307117 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.307915 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.308632 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.308930 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.309002 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.309114 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.309245 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.309314 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.309461 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.309483 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 18:58:25.314891 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.312294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51907 (local address 127.0.98.2:33829)
0929 18:58:25.312356 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.312358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.312536 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:58:25.312602 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.312603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.312613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.312661 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.312813 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.313263 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.314129 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.314338 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.314412 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.314634 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.314660 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.314721 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.314790 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.314807 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:58:25.314924 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.312232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33829 (local address 127.0.98.1:51907)
0929 18:58:25.312466 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.312475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.312550 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.312661 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.312662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.312663 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:25.312729 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.312730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.313357 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.313358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.314040 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.314043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.314421 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.314422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.314484 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.314503 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.314724 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.314726 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.314785 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.314828 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260929 18:58:25.317046 2554 raft_consensus.cc:1261] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Refusing update from remote peer 1012737b25e64cf1925d967744e151b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.317314 2631 consensus_queue.cc:1035] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.317463 2414 raft_consensus.cc:1261] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Refusing update from remote peer 1012737b25e64cf1925d967744e151b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:25.317708 2631 consensus_queue.cc:1035] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:25.325395 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.322567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39005 (local address 127.0.98.1:43063)
0929 18:58:25.322615 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:25.322617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:25.322624 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:25.322644 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:25.322645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:25.322655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:25.322709 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:25.322855 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.323618 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.324654 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.324944 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.325009 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.325175 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.325207 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.325256 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:25.325312 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:58:25.325329 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:58:25.325428 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:25.322432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.98.2:39005)
0929 18:58:25.322466 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:25.322478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:25.322518 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:25.322749 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:25.322750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:25.322751 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:25.322788 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.322789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.323705 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.323706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.324565 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:25.324568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:25.325042 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:25.325043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:25.325100 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:25.325104 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:25.325258 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:25.325261 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:25.325310 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:58:25.325356 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 18:58:25.325793 2650 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:58:26.846107 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:26.846202 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:26.847019 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Sending DeleteTablet for 3 replicas of tablet bb17b8c4618c44e2a3585e3c530e279f
I20260929 18:58:26.847326 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet bb17b8c4618c44e2a3585e3c530e279f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41190
I20260929 18:58:26.847386 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet bb17b8c4618c44e2a3585e3c530e279f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48740
I20260929 18:58:26.847411 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet bb17b8c4618c44e2a3585e3c530e279f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260929 18:58:26.847642 2701 tablet_replica.cc:330] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: stopping tablet replica
I20260929 18:58:26.847699 2701 raft_consensus.cc:2229] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:26.847733 2701 raft_consensus.cc:2258] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.847786 2702 tablet_replica.cc:330] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: stopping tablet replica
I20260929 18:58:26.847836 2702 raft_consensus.cc:2229] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:26.847911 2702 raft_consensus.cc:2258] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.847975 2701 ts_tablet_manager.cc:1905] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:26.848184 2702 ts_tablet_manager.cc:1905] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:26.848039 2703 tablet_replica.cc:330] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: stopping tablet replica
I20260929 18:58:26.850131 2702 ts_tablet_manager.cc:1918] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:58:26.850200 2702 log.cc:1196] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/bb17b8c4618c44e2a3585e3c530e279f
I20260929 18:58:26.850458 2702 ts_tablet_manager.cc:1939] T bb17b8c4618c44e2a3585e3c530e279f P 1012737b25e64cf1925d967744e151b9: Deleting consensus metadata
I20260929 18:58:26.850713 2703 raft_consensus.cc:2229] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:26.850771 2703 raft_consensus.cc:2258] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.850778 2101 catalog_manager.cc:4841] TS 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063): tablet bb17b8c4618c44e2a3585e3c530e279f (table test-table [id=6d5f0984c2234e568031f58225dd1714]) successfully deleted
I20260929 18:58:26.851012 2703 ts_tablet_manager.cc:1905] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:26.852660 2701 ts_tablet_manager.cc:1918] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:58:26.852757 2701 log.cc:1196] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/bb17b8c4618c44e2a3585e3c530e279f
I20260929 18:58:26.852986 2701 ts_tablet_manager.cc:1939] T bb17b8c4618c44e2a3585e3c530e279f P cedc8a46e33244feb0c2d464161b432d: Deleting consensus metadata
I20260929 18:58:26.853049 2703 ts_tablet_manager.cc:1918] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:58:26.853109 2703 log.cc:1196] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/bb17b8c4618c44e2a3585e3c530e279f
I20260929 18:58:26.853360 2703 ts_tablet_manager.cc:1939] T bb17b8c4618c44e2a3585e3c530e279f P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting consensus metadata
I20260929 18:58:26.853358 2102 catalog_manager.cc:4841] TS cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829): tablet bb17b8c4618c44e2a3585e3c530e279f (table test-table [id=6d5f0984c2234e568031f58225dd1714]) successfully deleted
I20260929 18:58:26.853698 2101 catalog_manager.cc:4841] TS 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065): tablet bb17b8c4618c44e2a3585e3c530e279f (table test-table [id=6d5f0984c2234e568031f58225dd1714]) successfully deleted
Sep 29 18:58:26 dist-test-slave-gfsd krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:58:26 dist-test-slave-gfsd krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:26.873790 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:26.865721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.62:39815)
0929 18:58:26.865870 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:26.865872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:26.866000 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:58:26.866121 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:26.866122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:26.866134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:26.866222 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:26.866828 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:26.867460 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:26.868273 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:26.868512 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:26.868605 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:26.871015 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:26.871016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:26.871016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:26.871033 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:26.872527 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:26.872894 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:26.872896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:26.872896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:26.873012 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:26.873257 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:26.873258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:26.873259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:26.873374 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:26.873487 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:26.873585 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:58:26.873676 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260929 18:58:26.875365 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:26.875547 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:26.877116 2254 tablet_service.cc:1467] Processing CreateTablet for tablet ada18c963cd84a6d815c71eb39f39502 (DEFAULT_TABLE table=test-table [id=29186200dee645ff8e0de4c8373500dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.877156 2534 tablet_service.cc:1467] Processing CreateTablet for tablet ada18c963cd84a6d815c71eb39f39502 (DEFAULT_TABLE table=test-table [id=29186200dee645ff8e0de4c8373500dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.877238 2390 tablet_service.cc:1467] Processing CreateTablet for tablet ada18c963cd84a6d815c71eb39f39502 (DEFAULT_TABLE table=test-table [id=29186200dee645ff8e0de4c8373500dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.877281 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada18c963cd84a6d815c71eb39f39502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.877281 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada18c963cd84a6d815c71eb39f39502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.877360 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada18c963cd84a6d815c71eb39f39502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.878319 2712 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Bootstrap starting.
I20260929 18:58:26.878731 2714 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Bootstrap starting.
I20260929 18:58:26.878852 2712 tablet_bootstrap.cc:654] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.878875 2713 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Bootstrap starting.
I20260929 18:58:26.879134 2714 tablet_bootstrap.cc:654] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.879240 2713 tablet_bootstrap.cc:654] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.879745 2712 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: No bootstrap required, opened a new log
I20260929 18:58:26.879765 2714 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: No bootstrap required, opened a new log
I20260929 18:58:26.879804 2714 ts_tablet_manager.cc:1397] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:26.879803 2712 ts_tablet_manager.cc:1397] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:26.879930 2714 raft_consensus.cc:348] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.879946 2712 raft_consensus.cc:348] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.880008 2714 raft_consensus.cc:374] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.880030 2714 raft_consensus.cc:729] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.880031 2712 raft_consensus.cc:374] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.880054 2712 raft_consensus.cc:729] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7cbda3010b4fc2bce7b6ec2def0dd5, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.880079 2714 consensus_queue.cc:260] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.880118 2712 consensus_queue.cc:260] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.880173 2714 ts_tablet_manager.cc:1428] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:26.880264 2712 ts_tablet_manager.cc:1428] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:26.880752 2713 tablet_bootstrap.cc:492] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: No bootstrap required, opened a new log
I20260929 18:58:26.880844 2713 ts_tablet_manager.cc:1397] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:26.881062 2713 raft_consensus.cc:348] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.881181 2713 raft_consensus.cc:374] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.881232 2713 raft_consensus.cc:729] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.881287 2713 consensus_queue.cc:260] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.881386 2713 ts_tablet_manager.cc:1428] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:26.900573 2682 raft_consensus.cc:482] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:26.900650 2682 raft_consensus.cc:504] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.900789 2682 leader_election.cc:288] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:26.901005 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada18c963cd84a6d815c71eb39f39502" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9" is_pre_election: true
I20260929 18:58:26.901008 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada18c963cd84a6d815c71eb39f39502" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" is_pre_election: true
I20260929 18:58:26.901106 2274 raft_consensus.cc:2454] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:26.901104 2554 raft_consensus.cc:2454] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 0.
I20260929 18:58:26.901290 2345 leader_election.cc:302] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:26.901440 2682 raft_consensus.cc:2781] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:26.901500 2682 raft_consensus.cc:482] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:26.901517 2682 raft_consensus.cc:3037] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:26.902302 2682 raft_consensus.cc:504] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.902446 2682 leader_election.cc:288] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Requested vote from peers 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:26.902601 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada18c963cd84a6d815c71eb39f39502" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1012737b25e64cf1925d967744e151b9"
I20260929 18:58:26.902647 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada18c963cd84a6d815c71eb39f39502" candidate_uuid: "cedc8a46e33244feb0c2d464161b432d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
I20260929 18:58:26.902696 2274 raft_consensus.cc:3037] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:26.902710 2554 raft_consensus.cc:3037] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:26.903385 2274 raft_consensus.cc:2454] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:26.903404 2554 raft_consensus.cc:2454] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedc8a46e33244feb0c2d464161b432d in term 1.
I20260929 18:58:26.903586 2345 leader_election.cc:302] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:26.903709 2682 raft_consensus.cc:2781] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:26.903759 2682 raft_consensus.cc:686] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 1 LEADER]: Becoming Leader. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Running, Role: LEADER
I20260929 18:58:26.903810 2682 consensus_queue.cc:237] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.904387 2116 catalog_manager.cc:5495] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d reported cstate change: term changed from 0 to 1, leader changed from <none> to cedc8a46e33244feb0c2d464161b432d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cedc8a46e33244feb0c2d464161b432d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:26.911099 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:26.908827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.98.2:33829)
0929 18:58:26.908964 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:26.908966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:26.908972 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:26.908990 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:26.908991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:26.909000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:26.909054 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:26.909143 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:26.909619 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:26.910391 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:26.910605 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:26.910670 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:26.910787 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:26.910815 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:26.910855 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:26.910903 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:58:26.910972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260929 18:58:26.913709 2554 raft_consensus.cc:1261] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:26.913743 2274 raft_consensus.cc:1261] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Refusing update from remote peer cedc8a46e33244feb0c2d464161b432d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:26.913954 2659 consensus_queue.cc:1035] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:26.914063 2687 consensus_queue.cc:1035] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:26.924979 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:26.925065 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:26.925693 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Sending DeleteTablet for 3 replicas of tablet ada18c963cd84a6d815c71eb39f39502
I20260929 18:58:26.926007 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet ada18c963cd84a6d815c71eb39f39502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260929 18:58:26.926087 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet ada18c963cd84a6d815c71eb39f39502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48740
I20260929 18:58:26.926091 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet ada18c963cd84a6d815c71eb39f39502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41190
I20260929 18:58:26.926122 2703 tablet_replica.cc:330] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: stopping tablet replica
I20260929 18:58:26.926178 2702 tablet_replica.cc:330] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: stopping tablet replica
I20260929 18:58:26.926182 2703 raft_consensus.cc:2229] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:26.926206 2701 tablet_replica.cc:330] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: stopping tablet replica
I20260929 18:58:26.926222 2702 raft_consensus.cc:2229] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:26.926230 2703 raft_consensus.cc:2258] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.926257 2702 raft_consensus.cc:2258] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.926270 2701 raft_consensus.cc:2229] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:26.926340 2701 raft_consensus.cc:2258] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:26.926616 2702 ts_tablet_manager.cc:1905] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:26.926656 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:58:26.926740 2701 ts_tablet_manager.cc:1905] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:58:26.926800 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:26.928095 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 2611d349feae49b2aa89a4aa496c6117 (DEFAULT_TABLE table=test-table [id=b2f552e7abf14315b533349e4ee73262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.928141 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 2611d349feae49b2aa89a4aa496c6117 (DEFAULT_TABLE table=test-table [id=b2f552e7abf14315b533349e4ee73262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.928205 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2611d349feae49b2aa89a4aa496c6117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.928258 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2611d349feae49b2aa89a4aa496c6117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.928382 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 2611d349feae49b2aa89a4aa496c6117 (DEFAULT_TABLE table=test-table [id=b2f552e7abf14315b533349e4ee73262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:26.928489 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2611d349feae49b2aa89a4aa496c6117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:26.928565 2701 ts_tablet_manager.cc:1918] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:26.928614 2701 log.cc:1196] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/ada18c963cd84a6d815c71eb39f39502
I20260929 18:58:26.928925 2701 ts_tablet_manager.cc:1939] T ada18c963cd84a6d815c71eb39f39502 P cedc8a46e33244feb0c2d464161b432d: Deleting consensus metadata
I20260929 18:58:26.929315 2102 catalog_manager.cc:4841] TS cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829): tablet ada18c963cd84a6d815c71eb39f39502 (table test-table [id=29186200dee645ff8e0de4c8373500dd]) successfully deleted
I20260929 18:58:26.929486 2702 ts_tablet_manager.cc:1918] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:26.929539 2702 log.cc:1196] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/ada18c963cd84a6d815c71eb39f39502
I20260929 18:58:26.929780 2702 ts_tablet_manager.cc:1939] T ada18c963cd84a6d815c71eb39f39502 P 1012737b25e64cf1925d967744e151b9: Deleting consensus metadata
I20260929 18:58:26.929736 2714 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Bootstrap starting.
I20260929 18:58:26.930109 2714 tablet_bootstrap.cc:654] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.930117 2101 catalog_manager.cc:4841] TS 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063): tablet ada18c963cd84a6d815c71eb39f39502 (table test-table [id=29186200dee645ff8e0de4c8373500dd]) successfully deleted
I20260929 18:58:26.930142 2712 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Bootstrap starting.
I20260929 18:58:26.930480 2703 ts_tablet_manager.cc:1905] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:26.930539 2712 tablet_bootstrap.cc:654] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.931093 2712 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: No bootstrap required, opened a new log
I20260929 18:58:26.931145 2712 ts_tablet_manager.cc:1397] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:26.931322 2712 raft_consensus.cc:348] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.931414 2712 raft_consensus.cc:374] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.931440 2712 raft_consensus.cc:729] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7cbda3010b4fc2bce7b6ec2def0dd5, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.931504 2712 consensus_queue.cc:260] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.931623 2712 ts_tablet_manager.cc:1428] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:26.931774 2703 ts_tablet_manager.cc:1918] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:26.931813 2703 log.cc:1196] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/ada18c963cd84a6d815c71eb39f39502
I20260929 18:58:26.931981 2703 ts_tablet_manager.cc:1939] T ada18c963cd84a6d815c71eb39f39502 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting consensus metadata
I20260929 18:58:26.932150 2713 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9: Bootstrap starting.
I20260929 18:58:26.932216 2714 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: No bootstrap required, opened a new log
I20260929 18:58:26.932265 2714 ts_tablet_manager.cc:1397] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 18:58:26.932363 2714 raft_consensus.cc:348] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.932438 2714 raft_consensus.cc:374] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.932461 2714 raft_consensus.cc:729] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedc8a46e33244feb0c2d464161b432d, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.932518 2714 consensus_queue.cc:260] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.932552 2713 tablet_bootstrap.cc:654] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:26.932627 2714 ts_tablet_manager.cc:1428] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:26.933058 2101 catalog_manager.cc:4841] TS 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065): tablet ada18c963cd84a6d815c71eb39f39502 (table test-table [id=29186200dee645ff8e0de4c8373500dd]) successfully deleted
I20260929 18:58:26.933593 2713 tablet_bootstrap.cc:492] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9: No bootstrap required, opened a new log
I20260929 18:58:26.933642 2713 ts_tablet_manager.cc:1397] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:26.933784 2713 raft_consensus.cc:348] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.933859 2713 raft_consensus.cc:374] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:26.933940 2713 raft_consensus.cc:729] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Initialized, Role: FOLLOWER
I20260929 18:58:26.934003 2713 consensus_queue.cc:260] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:26.934120 2713 ts_tablet_manager.cc:1428] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:27.081208 2678 raft_consensus.cc:482] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:27.081324 2678 raft_consensus.cc:504] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:27.081578 2678 leader_election.cc:288] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:27.081808 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2611d349feae49b2aa89a4aa496c6117" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc8a46e33244feb0c2d464161b432d" is_pre_election: true
I20260929 18:58:27.081817 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2611d349feae49b2aa89a4aa496c6117" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" is_pre_election: true
I20260929 18:58:27.081983 2414 raft_consensus.cc:2454] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 0.
I20260929 18:58:27.081977 2554 raft_consensus.cc:2454] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 0.
I20260929 18:58:27.082187 2206 leader_election.cc:302] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, cedc8a46e33244feb0c2d464161b432d; no voters:
I20260929 18:58:27.082317 2678 raft_consensus.cc:2781] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:27.082391 2678 raft_consensus.cc:482] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:27.082432 2678 raft_consensus.cc:3037] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:27.083194 2678 raft_consensus.cc:504] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:27.083419 2678 leader_election.cc:288] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 election: Requested vote from peers cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829), 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065)
I20260929 18:58:27.083587 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2611d349feae49b2aa89a4aa496c6117" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5"
I20260929 18:58:27.083690 2554 raft_consensus.cc:3037] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:27.083635 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2611d349feae49b2aa89a4aa496c6117" candidate_uuid: "1012737b25e64cf1925d967744e151b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedc8a46e33244feb0c2d464161b432d"
I20260929 18:58:27.083740 2414 raft_consensus.cc:3037] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:27.084290 2554 raft_consensus.cc:2454] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 1.
I20260929 18:58:27.084429 2414 raft_consensus.cc:2454] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1012737b25e64cf1925d967744e151b9 in term 1.
I20260929 18:58:27.084462 2205 leader_election.cc:302] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1012737b25e64cf1925d967744e151b9, 2a7cbda3010b4fc2bce7b6ec2def0dd5; no voters:
I20260929 18:58:27.084592 2678 raft_consensus.cc:2781] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:27.084676 2678 raft_consensus.cc:686] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1012737b25e64cf1925d967744e151b9, State: Running, Role: LEADER
I20260929 18:58:27.084731 2678 consensus_queue.cc:237] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260929 18:58:27.085340 2117 catalog_manager.cc:5495] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1012737b25e64cf1925d967744e151b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1012737b25e64cf1925d967744e151b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1012737b25e64cf1925d967744e151b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:27.121393 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:27.118951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.98.1:43063)
0929 18:58:27.119126 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:27.119128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:27.119134 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:27.119154 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:27.119155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:27.119164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:27.119221 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:27.119390 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:27.119892 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:27.120623 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:27.120883 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:27.121011 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:27.121154 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:27.121181 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:27.121220 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:27.121269 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:58:27.121285 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
I20260929 18:58:27.123541 2554 raft_consensus.cc:1261] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Refusing update from remote peer 1012737b25e64cf1925d967744e151b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:27.123545 2414 raft_consensus.cc:1261] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Refusing update from remote peer 1012737b25e64cf1925d967744e151b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:27.123755 2678 consensus_queue.cc:1035] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cedc8a46e33244feb0c2d464161b432d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:27.123868 2696 consensus_queue.cc:1035] T 2611d349feae49b2aa89a4aa496c6117 P 1012737b25e64cf1925d967744e151b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7cbda3010b4fc2bce7b6ec2def0dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:58:27.128475 2685 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:58:28.645123 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:28.645248 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:28.646080 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fdd544775c245b4aaebd684121aca92: Sending DeleteTablet for 3 replicas of tablet 2611d349feae49b2aa89a4aa496c6117
I20260929 18:58:28.646317 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 2611d349feae49b2aa89a4aa496c6117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41190
I20260929 18:58:28.646376 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 2611d349feae49b2aa89a4aa496c6117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48740
I20260929 18:58:28.646462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2189
I20260929 18:58:28.646507 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 2611d349feae49b2aa89a4aa496c6117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260929 18:58:28.647107 2758 tablet_replica.cc:330] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: stopping tablet replica
I20260929 18:58:28.647172 2758 raft_consensus.cc:2229] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:28.647220 2758 raft_consensus.cc:2258] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:28.647336 2756 tablet_replica.cc:330] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: stopping tablet replica
I20260929 18:58:28.647435 2756 raft_consensus.cc:2229] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:28.647634 2758 ts_tablet_manager.cc:1905] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:28.649026 2756 raft_consensus.cc:2258] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:28.649331 2756 ts_tablet_manager.cc:1905] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:28.649415 2758 ts_tablet_manager.cc:1918] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:58:28.649492 2758 log.cc:1196] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/2611d349feae49b2aa89a4aa496c6117
I20260929 18:58:28.649806 2758 ts_tablet_manager.cc:1939] T 2611d349feae49b2aa89a4aa496c6117 P 2a7cbda3010b4fc2bce7b6ec2def0dd5: Deleting consensus metadata
I20260929 18:58:28.650144 2101 catalog_manager.cc:4841] TS 2a7cbda3010b4fc2bce7b6ec2def0dd5 (127.0.98.3:36065): tablet 2611d349feae49b2aa89a4aa496c6117 (table test-table [id=b2f552e7abf14315b533349e4ee73262]) successfully deleted
I20260929 18:58:28.650848 2756 ts_tablet_manager.cc:1918] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:58:28.650916 2756 log.cc:1196] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/2611d349feae49b2aa89a4aa496c6117
I20260929 18:58:28.651125 2756 ts_tablet_manager.cc:1939] T 2611d349feae49b2aa89a4aa496c6117 P cedc8a46e33244feb0c2d464161b432d: Deleting consensus metadata
I20260929 18:58:28.651513 2102 catalog_manager.cc:4841] TS cedc8a46e33244feb0c2d464161b432d (127.0.98.2:33829): tablet 2611d349feae49b2aa89a4aa496c6117 (table test-table [id=b2f552e7abf14315b533349e4ee73262]) successfully deleted
W20260929 18:58:28.653596 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43063 (error 108)
I20260929 18:58:28.653836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2329
I20260929 18:58:28.655157 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:28.654053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:48762)
0929 18:58:28.655054 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:28.655108 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":62,"threads_started":1}
W20260929 18:58:28.655315 2101 catalog_manager.cc:4584] TS 1012737b25e64cf1925d967744e151b9 (127.0.98.1:43063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2611d349feae49b2aa89a4aa496c6117: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
I20260929 18:58:28.660101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2469
I20260929 18:58:28.664853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2085
2026-09-29T18:58:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5635 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:28 dist-test-slave-gfsd krb5kdc[2763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:28 dist-test-slave-gfsd krb5kdc[2763](info): set up 2 sockets
Sep 29 18:58:28 dist-test-slave-gfsd krb5kdc[2763](info): commencing operation
krb5kdc: starting...
W20260929 18:58:29.707142 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:29 dist-test-slave-gfsd krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:29.826704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:29.906186 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:29.906386 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:29.906409 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:29.907514 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:29.907547 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:29.907562 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:29.907574 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:29.907586 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:29.908960 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:29.909140 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:29.909310 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:29.914387 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:29.914402 2776 server_base.cc:1034] running on GCE node
W20260929 18:58:29.914387 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:29.914403 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:29.914863 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:29.915165 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:29.916318 2776 hybrid_clock.cc:648] HybridClock initialized: now 1790708309916278 us; error 92 us; skew 500 ppm
Sep 29 18:58:29 dist-test-slave-gfsd krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:29.918752 2776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:29.919189 2776 webserver.cc:460] Webserver started at http://127.0.98.62:45521/ using document root <none> and password file <none>
I20260929 18:58:29.919435 2776 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:29.919503 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:29.919649 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:29.920677 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "027ab204cc624238b96f31cdc30fcc48"
format_stamp: "Formatted at 2026-09-29 18:58:29 on dist-test-slave-gfsd"
server_key: "58d49f9cc0a2d96feb70f2c4cdbab506"
server_key_iv: "f7c1e504a952c235e06f7757383df8e2"
server_key_version: "encryptionkey@0"
I20260929 18:58:29.921053 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "027ab204cc624238b96f31cdc30fcc48"
format_stamp: "Formatted at 2026-09-29 18:58:29 on dist-test-slave-gfsd"
server_key: "58d49f9cc0a2d96feb70f2c4cdbab506"
server_key_iv: "f7c1e504a952c235e06f7757383df8e2"
server_key_version: "encryptionkey@0"
I20260929 18:58:29.922402 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:29.923167 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:29.923364 2776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:29.923468 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "027ab204cc624238b96f31cdc30fcc48"
format_stamp: "Formatted at 2026-09-29 18:58:29 on dist-test-slave-gfsd"
server_key: "58d49f9cc0a2d96feb70f2c4cdbab506"
server_key_iv: "f7c1e504a952c235e06f7757383df8e2"
server_key_version: "encryptionkey@0"
I20260929 18:58:29.923548 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:29.937588 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:29.939579 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:29.939728 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:29.944015 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36961
I20260929 18:58:29.944077 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36961 every 8 connection(s)
I20260929 18:58:29.944576 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:29.945247 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:29.947680 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48: Bootstrap starting.
I20260929 18:58:29.948237 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:29.948523 2844 log.cc:824] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:29.949259 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48: No bootstrap required, opened a new log
I20260929 18:58:29.950805 2844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } }
I20260929 18:58:29.950937 2844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:29.950958 2844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027ab204cc624238b96f31cdc30fcc48, State: Initialized, Role: FOLLOWER
I20260929 18:58:29.951112 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } }
I20260929 18:58:29.951211 2844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:29.951253 2844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:29.951313 2844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:29.951349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2776
I20260929 18:58:29.951447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:29.951618 392 external_mini_cluster.cc:1428] Setting key 72feb5b6ea88f345c15ad8eee7909f2c
I20260929 18:58:29.952456 2844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } }
I20260929 18:58:29.952548 2844 leader_election.cc:302] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 027ab204cc624238b96f31cdc30fcc48; no voters:
I20260929 18:58:29.952733 2844 leader_election.cc:288] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:29.952757 2849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:29.953001 2849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [term 1 LEADER]: Becoming Leader. State: Replica: 027ab204cc624238b96f31cdc30fcc48, State: Running, Role: LEADER
I20260929 18:58:29.952986 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:29.953188 2849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } }
I20260929 18:58:29.954239 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 027ab204cc624238b96f31cdc30fcc48. Latest consensus state: current_term: 1 leader_uuid: "027ab204cc624238b96f31cdc30fcc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } } }
I20260929 18:58:29.954337 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:29.954598 2864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:29.954598 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "027ab204cc624238b96f31cdc30fcc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027ab204cc624238b96f31cdc30fcc48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36961 } } }
I20260929 18:58:29.955199 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:29.955678 2864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:29.957304 2864 catalog_manager.cc:1347] Generated new cluster ID: 4b6dc7af77da4fb8b45f8287dac44003
I20260929 18:58:29.957360 2864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:29 dist-test-slave-gfsd krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:29.962700 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:29.952113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.0.98.62:36961)
0929 18:58:29.952291 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:29.952299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:29.952323 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:58:29.952412 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:29.952417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:29.952464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:29.952616 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:29.953442 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:29.954832 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:29.955747 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:29.956149 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:29.956258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:29.959020 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:29.959025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:29.959025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:29.959037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:29.961039 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:29.961428 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:29.961432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:29.961432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:29.961564 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:29.961767 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:29.961768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:29.961768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:29.962121 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:29.962284 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:29.962359 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:58:29.962496 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260929 18:58:29.979086 2864 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:29.979897 2864 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:29.985985 2864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 027ab204cc624238b96f31cdc30fcc48: Generated new TSK 0
I20260929 18:58:29.986246 2864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:30.022611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:30.127486 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:30.127693 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:30.127717 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:30.128873 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:30.128913 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:30.128971 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:30.130560 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:30.130807 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:30.131054 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:30.136132 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:30.136132 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:30.136191 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:30.136324 2870 server_base.cc:1034] running on GCE node
I20260929 18:58:30.136641 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:30.136842 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:30.137960 2870 hybrid_clock.cc:648] HybridClock initialized: now 1790708310137938 us; error 40 us; skew 500 ppm
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:30.140686 2870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:30.141172 2870 webserver.cc:460] Webserver started at http://127.0.98.1:34007/ using document root <none> and password file <none>
I20260929 18:58:30.141400 2870 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:30.141469 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:30.141628 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:30.142648 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c5adc25319548f4875411c137e8b777"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "30d26d081a58e7761d9a345e004f731b"
server_key_iv: "c1df4f70b97f39538d8c3a5ec0d2b705"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.143014 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c5adc25319548f4875411c137e8b777"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "30d26d081a58e7761d9a345e004f731b"
server_key_iv: "c1df4f70b97f39538d8c3a5ec0d2b705"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.144367 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:58:30.145197 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.145401 2870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:30.145504 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "6c5adc25319548f4875411c137e8b777"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "30d26d081a58e7761d9a345e004f731b"
server_key_iv: "c1df4f70b97f39538d8c3a5ec0d2b705"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.145589 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:30.164330 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:30.166684 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:30.166852 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:30.167150 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:30.167486 2870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:30.167548 2870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.167584 2870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:30.167631 2870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.173488 2870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43315
I20260929 18:58:30.173539 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43315 every 8 connection(s)
I20260929 18:58:30.173846 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:30.178630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 2870
I20260929 18:58:30.178761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:30.178949 392 external_mini_cluster.cc:1428] Setting key 1af847223072cd5c37b01e742a655931
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:30.184262 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.174601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33185 (local address 127.0.98.62:36961)
0929 18:58:30.174683 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:30.174684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:30.174851 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:58:30.175916 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:30.175917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:30.175931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:30.176023 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:30.176629 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.177160 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.178144 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.178331 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.178386 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.180977 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:30.180979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:30.180980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:30.180996 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:30.182722 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.183087 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.183088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.183089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.183152 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.183478 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.183479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.183479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.183632 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:30.183714 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:30.184079 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 18:58:30.184171 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:58:30.184425 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.174560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36961 (local address 127.0.98.1:33185)
0929 18:58:30.174715 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:30.174757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:30.175762 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:30.176085 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:30.176089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:30.176335 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:30.176537 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.176541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.177249 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.177250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.178062 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.178065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.178404 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.178405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.178503 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.179093 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:30.179107 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:30.180880 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:30.182797 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.182802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.182804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.183011 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.183223 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.183224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.183224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.183393 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.183728 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:30.183733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:30.183851 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:58:30.184023 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:58:30.184187 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260929 18:58:30.185031 2999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36961
I20260929 18:58:30.185153 2999 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:30.185333 2999 heartbeater.cc:511] Master 127.0.98.62:36961 requested a full tablet report, sending...
I20260929 18:58:30.185940 2808 ts_manager.cc:194] Registered new tserver with Master: 6c5adc25319548f4875411c137e8b777 (127.0.98.1:43315)
I20260929 18:58:30.186555 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33185
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:30.205794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:30.288368 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:30.288612 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:30.288653 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:30.289844 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:30.289913 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:30.289969 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:30.291538 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:30.291795 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:30.292150 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:30.297465 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:30.297492 3004 server_base.cc:1034] running on GCE node
W20260929 18:58:30.297441 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:30.297500 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:30.297924 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:30.298133 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:30.299327 3004 hybrid_clock.cc:648] HybridClock initialized: now 1790708310299316 us; error 30 us; skew 500 ppm
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:30.301877 3004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:30.302289 3004 webserver.cc:460] Webserver started at http://127.0.98.2:35059/ using document root <none> and password file <none>
I20260929 18:58:30.302507 3004 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:30.302551 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:30.302683 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:30.303766 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "4482ac29937e4c75b671e3148f48d930"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "1d48916b47a1ad8424aa6d5796dae2bb"
server_key_iv: "97b1fd28070af76b18a2598b207ffaa6"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.304127 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4482ac29937e4c75b671e3148f48d930"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "1d48916b47a1ad8424aa6d5796dae2bb"
server_key_iv: "97b1fd28070af76b18a2598b207ffaa6"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.305461 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:30.306288 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.306497 3004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:30.306587 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "4482ac29937e4c75b671e3148f48d930"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "1d48916b47a1ad8424aa6d5796dae2bb"
server_key_iv: "97b1fd28070af76b18a2598b207ffaa6"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.306658 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:30.340225 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:30.342375 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:30.342525 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:30.342772 3004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:30.343086 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:30.343142 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.343185 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:30.343214 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.348990 3004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41759
I20260929 18:58:30.349027 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41759 every 8 connection(s)
I20260929 18:58:30.349346 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:30.351046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3004
I20260929 18:58:30.351127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:30.351389 392 external_mini_cluster.cc:1428] Setting key 3762bb416d8b87ae0e80477dbcf0c891
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:30.360304 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.350196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52537 (local address 127.0.98.62:36961)
0929 18:58:30.350282 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:30.350285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:30.350456 (+ 171us) server_negotiation.cc:407] Connection header received
0929 18:58:30.351534 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:30.351535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:30.351549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:30.351599 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:30.352234 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.352819 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.354197 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.354517 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.354591 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.357067 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:30.357068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:30.357069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:30.357077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:30.358789 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.359242 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.359244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.359244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.359343 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.359629 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.359630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.359631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.359774 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:30.359853 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:30.360099 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:58:30.360187 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:58:30.360332 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.350159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36961 (local address 127.0.98.2:52537)
0929 18:58:30.350322 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:30.350364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:30.351412 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:30.351669 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:30.351673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:30.351935 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:30.352150 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.352156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.352908 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.352909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.354123 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.354125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.354622 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.354623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.354795 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.355328 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:30.355338 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:30.356962 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:30.358892 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.358897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.358899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.359141 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.359435 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.359436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.359436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.359544 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.359847 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:30.359851 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:30.359966 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:58:30.360052 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:58:30.360172 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260929 18:58:30.360949 3133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36961
I20260929 18:58:30.361064 3133 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:58:30.361276 3133 heartbeater.cc:511] Master 127.0.98.62:36961 requested a full tablet report, sending...
I20260929 18:58:30.361734 2808 ts_manager.cc:194] Registered new tserver with Master: 4482ac29937e4c75b671e3148f48d930 (127.0.98.2:41759)
I20260929 18:58:30.362115 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:30.379007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:30.460386 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:30.460594 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:30.460618 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:30.461697 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:30.461731 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:30.461757 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:30.463156 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:30.463392 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:30.463575 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:30.468497 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:30.468670 3138 server_base.cc:1034] running on GCE node
W20260929 18:58:30.468497 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:30.468485 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:30.469038 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:30.469308 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:30.470472 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790708310470454 us; error 31 us; skew 500 ppm
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:30.472918 3138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:30.473462 3138 webserver.cc:460] Webserver started at http://127.0.98.3:43009/ using document root <none> and password file <none>
I20260929 18:58:30.473716 3138 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:30.473783 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:30.473924 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:30.474992 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "fdf8ebc4a88f47f3a2ccd6930318c48d"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "0ff53e86d18e0c8562824b0c52ae1edf"
server_key_iv: "bfd5629a85ad3c4eac8560edc756524d"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.475450 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fdf8ebc4a88f47f3a2ccd6930318c48d"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "0ff53e86d18e0c8562824b0c52ae1edf"
server_key_iv: "bfd5629a85ad3c4eac8560edc756524d"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.476856 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:30.477689 3153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.477895 3138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:30.477996 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "fdf8ebc4a88f47f3a2ccd6930318c48d"
format_stamp: "Formatted at 2026-09-29 18:58:30 on dist-test-slave-gfsd"
server_key: "0ff53e86d18e0c8562824b0c52ae1edf"
server_key_iv: "bfd5629a85ad3c4eac8560edc756524d"
server_key_version: "encryptionkey@0"
I20260929 18:58:30.478085 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:30.493160 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:30.495273 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:30.495476 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:30.495704 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:30.496011 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:30.496063 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.496104 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:30.496142 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:30.501919 3138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33701
I20260929 18:58:30.501960 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33701 every 8 connection(s)
I20260929 18:58:30.502302 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:30.503755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3138
I20260929 18:58:30.503842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:30.504048 392 external_mini_cluster.cc:1428] Setting key 25df14acfba426af48a86126788434f5
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:30.511972 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.502992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57793 (local address 127.0.98.62:36961)
0929 18:58:30.503055 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:30.503057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:30.503319 (+ 262us) server_negotiation.cc:407] Connection header received
0929 18:58:30.504701 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:30.504702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:30.504714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:30.504763 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:30.505434 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.505921 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.506907 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.507111 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.507168 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.509224 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:30.509225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:30.509225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:30.509236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:30.510683 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.511144 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.511145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.511146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.511196 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:30.511456 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:30.511456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:30.511456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:30.511556 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:30.511609 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:30.511849 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:58:30.511909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:58:30.512207 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:30.503008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36961 (local address 127.0.98.3:57793)
0929 18:58:30.503179 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:30.503213 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:30.504581 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:30.504857 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:30.504862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:30.505165 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:30.505355 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.505359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.506030 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.506030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.506828 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:30.506830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:30.507187 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:30.507188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:30.507330 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:30.507802 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:30.507810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:30.509148 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:30.510750 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.510755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.510756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.511070 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.511254 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:30.511254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:30.511255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:30.511399 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:30.511616 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:30.511620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:30.511730 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:58:30.511807 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:58:30.511921 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260929 18:58:30.512717 3267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36961
I20260929 18:58:30.512792 3267 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:30.512970 3267 heartbeater.cc:511] Master 127.0.98.62:36961 requested a full tablet report, sending...
I20260929 18:58:30.513367 2808 ts_manager.cc:194] Registered new tserver with Master: fdf8ebc4a88f47f3a2ccd6930318c48d (127.0.98.3:33701)
I20260929 18:58:30.513762 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57793
I20260929 18:58:30.515623 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:58:30.526228 2848 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:30.525295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.0.98.62:36961)
0929 18:58:30.525382 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:30.525385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:30.525505 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:58:30.525577 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:30.525578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:30.525588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:30.525640 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:30.526175 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50424: BlockingRecv error: recv got EOF from 127.0.0.1:50424 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:58:30.527407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2870
I20260929 18:58:30.532395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3004
I20260929 18:58:30.536334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3138
I20260929 18:58:30.540450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 2776
2026-09-29T18:58:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1876 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[3279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[3279](info): set up 2 sockets
Sep 29 18:58:30 dist-test-slave-gfsd krb5kdc[3279](info): commencing operation
krb5kdc: starting...
W20260929 18:58:31.575263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:31.707816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:31.787060 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:31.787302 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:31.787358 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:31.788749 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:31.788811 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:31.788836 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:31.788867 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:31.788897 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:31.790403 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:31.790670 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:31.790904 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:31.795547 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:31.795598 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:31.795739 3292 server_base.cc:1034] running on GCE node
W20260929 18:58:31.795855 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:31.796164 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:31.796452 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:31.797622 3292 hybrid_clock.cc:648] HybridClock initialized: now 1790708311797607 us; error 49 us; skew 500 ppm
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:31.803272 3292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:31.803740 3292 webserver.cc:460] Webserver started at http://127.0.98.62:38351/ using document root <none> and password file <none>
I20260929 18:58:31.803977 3292 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:31.804044 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:31.804189 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:31.805226 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "fa4583af87c343e3a6d4aa802f12584c"
format_stamp: "Formatted at 2026-09-29 18:58:31 on dist-test-slave-gfsd"
server_key: "7739192faf55d7c0da9e8b6e805ba36c"
server_key_iv: "bbea6fb1d99bb14e30a86f9a17a9425c"
server_key_version: "encryptionkey@0"
I20260929 18:58:31.805568 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa4583af87c343e3a6d4aa802f12584c"
format_stamp: "Formatted at 2026-09-29 18:58:31 on dist-test-slave-gfsd"
server_key: "7739192faf55d7c0da9e8b6e805ba36c"
server_key_iv: "bbea6fb1d99bb14e30a86f9a17a9425c"
server_key_version: "encryptionkey@0"
I20260929 18:58:31.806867 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:31.807751 3307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:31.808017 3292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:31.808125 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "fa4583af87c343e3a6d4aa802f12584c"
format_stamp: "Formatted at 2026-09-29 18:58:31 on dist-test-slave-gfsd"
server_key: "7739192faf55d7c0da9e8b6e805ba36c"
server_key_iv: "bbea6fb1d99bb14e30a86f9a17a9425c"
server_key_version: "encryptionkey@0"
I20260929 18:58:31.808197 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:31.825068 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:31.827160 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:31.827404 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:31.831252 3292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32883
I20260929 18:58:31.831385 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32883 every 8 connection(s)
I20260929 18:58:31.831714 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:31.832600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3292
I20260929 18:58:31.832697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:31.832779 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:31.832903 392 external_mini_cluster.cc:1428] Setting key 5d133305857ffdeaf0b4a144aa718946
I20260929 18:58:31.835779 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c: Bootstrap starting.
I20260929 18:58:31.836390 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:31.836673 3360 log.cc:824] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:31.837324 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c: No bootstrap required, opened a new log
I20260929 18:58:31.839254 3360 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260929 18:58:31.839515 3360 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:31.839551 3360 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4583af87c343e3a6d4aa802f12584c, State: Initialized, Role: FOLLOWER
I20260929 18:58:31.839684 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260929 18:58:31.839773 3360 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:31.839804 3360 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:31.839851 3360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:31.840575 3360 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260929 18:58:31.840662 3360 leader_election.cc:302] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4583af87c343e3a6d4aa802f12584c; no voters:
I20260929 18:58:31.840834 3360 leader_election.cc:288] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:31.840873 3365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:31.841071 3365 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [term 1 LEADER]: Becoming Leader. State: Replica: fa4583af87c343e3a6d4aa802f12584c, State: Running, Role: LEADER
I20260929 18:58:31.841104 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:31.841536 3365 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260929 18:58:31.843092 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4583af87c343e3a6d4aa802f12584c. Latest consensus state: current_term: 1 leader_uuid: "fa4583af87c343e3a6d4aa802f12584c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } } }
I20260929 18:58:31.843189 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:31.843407 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:31.843452 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4583af87c343e3a6d4aa802f12584c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4583af87c343e3a6d4aa802f12584c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } } }
I20260929 18:58:31.843513 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:31.843895 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:31.844540 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:31.833515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.62:32883)
0929 18:58:31.833713 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:31.833723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:31.833754 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:58:31.834301 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:31.834308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:31.834358 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:31.834530 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:31.835261 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:31.836271 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:31.837280 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:31.837775 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:31.837900 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:31.840522 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:31.840526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:31.840526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:31.840536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:31.842436 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:31.842864 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:31.842867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:31.842868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:31.842966 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:31.843219 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:31.843221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:31.843221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:31.843570 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:31.843718 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:31.844151 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:58:31.844280 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260929 18:58:31.845458 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:58:31.847141 3379 catalog_manager.cc:1347] Generated new cluster ID: 9ed141f7e4e04c5ba50691a230be5650
I20260929 18:58:31.847229 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:31.860042 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:31.860584 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:31.864095 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4583af87c343e3a6d4aa802f12584c: Generated new TSK 0
I20260929 18:58:31.864269 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:58:31.868834 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:31.864951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.98.62:32883)
0929 18:58:31.865019 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:31.865021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:31.865165 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:58:31.865248 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:31.865249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:31.865258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:31.865347 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:31.865769 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:31.866320 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:31.867429 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:31.867653 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:31.867712 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:31.868416 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:31.868417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:31.868417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:31.868586 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:31.868601 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:31.868607 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:31.868653 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:31.868730 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:58:31.868754 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:58:31.871297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3292
2026-09-29T18:58:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1327 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3393](info): set up 2 sockets
Sep 29 18:58:31 dist-test-slave-gfsd krb5kdc[3393](info): commencing operation
krb5kdc: starting...
W20260929 18:58:32.927175 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:33.043999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:33.122855 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:33.123080 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:33.123103 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:33.124207 3406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:33.124240 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:33.124254 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:33.124267 3406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:33.124279 3406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:33.125669 3406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:33.125869 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:33.126032 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:33.130897 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.130991 3406 server_base.cc:1034] running on GCE node
W20260929 18:58:33.130887 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.130882 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.131421 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:33.131705 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:33.132843 3406 hybrid_clock.cc:648] HybridClock initialized: now 1790708313132825 us; error 36 us; skew 500 ppm
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:33.135198 3406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:33.135722 3406 webserver.cc:460] Webserver started at http://127.0.98.62:44789/ using document root <none> and password file <none>
I20260929 18:58:33.135957 3406 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:33.136025 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:33.136205 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:33.137243 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "a331c9a85bf64a27b8f8a9554c69162f"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "aa6164c639718e0f0d8e8ec27846a023"
server_key_iv: "f8bd9a018c8dcdbc57c75fd2888e68ee"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.137606 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "a331c9a85bf64a27b8f8a9554c69162f"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "aa6164c639718e0f0d8e8ec27846a023"
server_key_iv: "f8bd9a018c8dcdbc57c75fd2888e68ee"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.138944 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:58:33.139835 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.140115 3406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:33.140221 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "a331c9a85bf64a27b8f8a9554c69162f"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "aa6164c639718e0f0d8e8ec27846a023"
server_key_iv: "f8bd9a018c8dcdbc57c75fd2888e68ee"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.140295 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:33.152559 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:33.154553 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:33.154726 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:33.158969 3406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33027
I20260929 18:58:33.159020 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33027 every 8 connection(s)
I20260929 18:58:33.159456 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:33.160414 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:33.162696 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f: Bootstrap starting.
I20260929 18:58:33.163319 3474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:33.163623 3474 log.cc:824] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:33.164332 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f: No bootstrap required, opened a new log
I20260929 18:58:33.165926 3474 raft_consensus.cc:348] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260929 18:58:33.166070 3474 raft_consensus.cc:374] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:33.166093 3474 raft_consensus.cc:729] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a331c9a85bf64a27b8f8a9554c69162f, State: Initialized, Role: FOLLOWER
I20260929 18:58:33.166231 3474 consensus_queue.cc:260] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260929 18:58:33.166306 3474 raft_consensus.cc:388] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:33.166327 3474 raft_consensus.cc:482] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:33.166392 3474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:33.167200 3474 raft_consensus.cc:504] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260929 18:58:33.167340 3474 leader_election.cc:302] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a331c9a85bf64a27b8f8a9554c69162f; no voters:
I20260929 18:58:33.167536 3474 leader_election.cc:288] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:33.167590 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:33.167773 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [term 1 LEADER]: Becoming Leader. State: Replica: a331c9a85bf64a27b8f8a9554c69162f, State: Running, Role: LEADER
I20260929 18:58:33.167795 3474 sys_catalog.cc:564] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:33.167866 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260929 18:58:33.168288 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a331c9a85bf64a27b8f8a9554c69162f. Latest consensus state: current_term: 1 leader_uuid: "a331c9a85bf64a27b8f8a9554c69162f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260929 18:58:33.168401 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:33.168568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3406
I20260929 18:58:33.168666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:33.168860 392 external_mini_cluster.cc:1428] Setting key 804b4eec135ba42527a4a4e8526c8a09
I20260929 18:58:33.168850 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:33.168277 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a331c9a85bf64a27b8f8a9554c69162f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a331c9a85bf64a27b8f8a9554c69162f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260929 18:58:33.169114 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:33.169435 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:33.171027 3486 catalog_manager.cc:1347] Generated new cluster ID: 50478275e826464ba07a24aafdf724c5
I20260929 18:58:33.171083 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:33.176998 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:33.177547 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:33.180851 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.169386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.62:33027)
0929 18:58:33.169590 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:33.169601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:33.169649 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:58:33.169776 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:33.169783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:33.169854 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:33.169985 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:33.170808 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.172279 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.173367 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.173999 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.174105 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.177108 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:33.177112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:33.177112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:33.177125 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:33.179210 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.179619 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.179622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.179622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.179730 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.179929 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.179929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.179930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.180221 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:33.180399 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:33.180467 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:58:33.180618 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260929 18:58:33.190074 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a331c9a85bf64a27b8f8a9554c69162f: Generated new TSK 0
I20260929 18:58:33.190255 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:33.218158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:33.298861 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:33.299101 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:33.299136 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:33.300328 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:33.300396 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:33.300441 3500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:33.301879 3500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:33.302145 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:33.302372 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:33.307444 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.307446 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.307436 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.307493 3500 server_base.cc:1034] running on GCE node
I20260929 18:58:33.307922 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:33.308135 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:33.309291 3500 hybrid_clock.cc:648] HybridClock initialized: now 1790708313309277 us; error 30 us; skew 500 ppm
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:33.311599 3500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:33.312038 3500 webserver.cc:460] Webserver started at http://127.0.98.1:36323/ using document root <none> and password file <none>
I20260929 18:58:33.312228 3500 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:33.312268 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:33.312352 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:33.313340 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "f11a173e0f0643e68cc33d13af7d5e4b"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "345c4c20a9f972e20b65f19fffa1d242"
server_key_iv: "785bbf92765f8bf7d2128320720642b7"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.313653 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f11a173e0f0643e68cc33d13af7d5e4b"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "345c4c20a9f972e20b65f19fffa1d242"
server_key_iv: "785bbf92765f8bf7d2128320720642b7"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.315122 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:58:33.316138 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.316349 3500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:33.316447 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "f11a173e0f0643e68cc33d13af7d5e4b"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "345c4c20a9f972e20b65f19fffa1d242"
server_key_iv: "785bbf92765f8bf7d2128320720642b7"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.316535 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:33.327023 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:33.329276 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:33.329433 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:33.329659 3500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:33.329957 3500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:33.330015 3500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.330055 3500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:33.330083 3500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.336195 3500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39789
I20260929 18:58:33.336251 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39789 every 8 connection(s)
I20260929 18:58:33.336596 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:33.343123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3500
I20260929 18:58:33.343223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:33.343431 392 external_mini_cluster.cc:1428] Setting key 1e76660a83d358c8214fdbb5d58bf868
I20260929 18:58:33.346475 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.337356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44591 (local address 127.0.98.62:33027)
0929 18:58:33.337433 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:33.337435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:33.337621 (+ 186us) server_negotiation.cc:407] Connection header received
0929 18:58:33.338227 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:33.338228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:33.338241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:33.338292 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:33.339266 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.339805 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.340834 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.341046 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.341124 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.343451 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:33.343453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:33.343453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:33.343461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:33.345009 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.345494 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.345495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.345495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.345545 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.345820 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.345821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.345822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.345942 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:33.346021 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:33.346295 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 18:58:33.346388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:33.346668 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.337280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.1:44591)
0929 18:58:33.337484 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:33.337534 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:33.338105 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:33.338365 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:33.338368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:33.338958 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:33.339153 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.339158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.339901 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.339903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.340727 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.340730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.341128 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.341129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.341260 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.341757 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:33.341765 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:33.343345 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:33.345091 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.345098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.345100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.345413 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.345619 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.345620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.345620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.345749 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.346012 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:33.346019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:33.346163 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:58:33.346273 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:58:33.346423 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260929 18:58:33.347220 3629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260929 18:58:33.347345 3629 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:33.347517 3629 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260929 18:58:33.348101 3438 ts_manager.cc:194] Registered new tserver with Master: f11a173e0f0643e68cc33d13af7d5e4b (127.0.98.1:39789)
I20260929 18:58:33.349114 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44591
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:33.368166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:33.450903 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:33.451103 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:33.451125 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:33.452327 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:33.452368 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:33.452395 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:33.453781 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:33.454022 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:33.454257 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:33.459378 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.459370 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.459369 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.459662 3634 server_base.cc:1034] running on GCE node
I20260929 18:58:33.459949 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:33.460202 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:33.461346 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790708313461323 us; error 37 us; skew 500 ppm
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:33.464068 3634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:33.464495 3634 webserver.cc:460] Webserver started at http://127.0.98.2:37557/ using document root <none> and password file <none>
I20260929 18:58:33.464735 3634 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:33.464802 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:33.464944 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:33.465986 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4fc872a1abe4a52a453c1b4795009e3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "21373e66a7722a1015945c3ef974d1ec"
server_key_iv: "dfc67fe8cac5e49371d861140992842f"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.466347 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4fc872a1abe4a52a453c1b4795009e3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "21373e66a7722a1015945c3ef974d1ec"
server_key_iv: "dfc67fe8cac5e49371d861140992842f"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.467731 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:58:33.468595 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.468765 3634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:33.468851 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "c4fc872a1abe4a52a453c1b4795009e3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "21373e66a7722a1015945c3ef974d1ec"
server_key_iv: "dfc67fe8cac5e49371d861140992842f"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.468927 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:33.480026 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:33.482458 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:33.482625 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:33.482879 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:33.483188 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:33.483239 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.483280 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:33.483330 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.489053 3634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33925
I20260929 18:58:33.489112 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33925 every 8 connection(s)
I20260929 18:58:33.489432 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:33.493045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3634
I20260929 18:58:33.493165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:33.493361 392 external_mini_cluster.cc:1428] Setting key 0b1d144c8d58003a3fbe7614d35efbc6
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:33.499516 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.490184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44911 (local address 127.0.98.62:33027)
0929 18:58:33.490227 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:33.490229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:33.490411 (+ 182us) server_negotiation.cc:407] Connection header received
0929 18:58:33.490911 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:33.490912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:33.490923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:33.490989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:33.492037 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.492622 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.493752 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.493945 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.494048 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.496435 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:33.496436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:33.496437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:33.496445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:33.498023 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.498482 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.498483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.498484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.498545 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.498812 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.498813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.498814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.498940 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:33.499023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:33.499331 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 18:58:33.499416 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:33.499709 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.490158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.2:44911)
0929 18:58:33.490293 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:33.490325 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:33.490799 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:33.491052 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:33.491056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:33.491676 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:33.491863 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.491868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.492708 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.492709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.493664 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.493667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.494049 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.494050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.494202 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.494702 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:33.494712 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:33.496312 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:33.498130 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.498137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.498139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.498375 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.498616 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.498617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.498617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.498738 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.499014 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:33.499020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:33.499167 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:58:33.499283 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:58:33.499477 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260929 18:58:33.500211 3763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260929 18:58:33.500306 3763 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:33.500505 3763 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260929 18:58:33.500900 3438 ts_manager.cc:194] Registered new tserver with Master: c4fc872a1abe4a52a453c1b4795009e3 (127.0.98.2:33925)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:58:33.501410 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:33.518723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:33.605230 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:33.605430 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:33.605453 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:33.606537 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:33.606570 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:33.606595 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:33.608002 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:33.608196 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:33.608356 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:33.613268 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:33.613276 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.613380 3768 server_base.cc:1034] running on GCE node
W20260929 18:58:33.613276 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:33.613677 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:33.613919 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:33.615131 3768 hybrid_clock.cc:648] HybridClock initialized: now 1790708313615123 us; error 20 us; skew 500 ppm
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:33.617637 3768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:33.618042 3768 webserver.cc:460] Webserver started at http://127.0.98.3:45235/ using document root <none> and password file <none>
I20260929 18:58:33.618250 3768 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:33.618315 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:33.618450 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:33.619556 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "1724ebca96614b798be8a630df6afea3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "81153ed11ad999744758425a6c2cab85"
server_key_iv: "d5b94d03a644414d68798f6cc732eb14"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.619915 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1724ebca96614b798be8a630df6afea3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "81153ed11ad999744758425a6c2cab85"
server_key_iv: "d5b94d03a644414d68798f6cc732eb14"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.621192 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:33.621903 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.622073 3768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:33.622164 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "1724ebca96614b798be8a630df6afea3"
format_stamp: "Formatted at 2026-09-29 18:58:33 on dist-test-slave-gfsd"
server_key: "81153ed11ad999744758425a6c2cab85"
server_key_iv: "d5b94d03a644414d68798f6cc732eb14"
server_key_version: "encryptionkey@0"
I20260929 18:58:33.622243 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:33.640614 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:33.642611 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:33.642755 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:33.642987 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:33.643348 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:33.643391 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.643437 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:33.643471 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:33.649215 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35281
I20260929 18:58:33.649291 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35281 every 8 connection(s)
I20260929 18:58:33.649590 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:33.653954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3768
I20260929 18:58:33.654179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:33.654445 392 external_mini_cluster.cc:1428] Setting key ab3f14fb30f3b35e6d726870460681af
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:33.659116 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.650307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45549 (local address 127.0.98.62:33027)
0929 18:58:33.650375 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:33.650377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:33.650569 (+ 192us) server_negotiation.cc:407] Connection header received
0929 18:58:33.651026 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:33.651027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:33.651037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:33.651106 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:33.652053 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.652536 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.653631 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.653917 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.654004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.656368 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:33.656369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:33.656370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:33.656378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:33.657735 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.658165 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.658166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.658166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.658224 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.658492 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.658493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.658493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.658600 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:33.658676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:33.658942 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 18:58:33.659021 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:58:33.659343 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.650290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.3:45549)
0929 18:58:33.650431 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:33.650478 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:33.650942 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:33.651160 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:33.651163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:33.651783 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:33.651972 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.651977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.652634 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.652635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.653522 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:33.653524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.654017 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.654018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:33.654116 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.654779 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:33.654789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:33.656268 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:33.657846 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.657852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.657855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.658086 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.658289 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:33.658289 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:33.658290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:33.658402 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:33.658666 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:33.658672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:33.658811 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:58:33.658920 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:58:33.659084 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260929 18:58:33.659754 3897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260929 18:58:33.659840 3897 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:33.660048 3897 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260929 18:58:33.660384 3438 ts_manager.cc:194] Registered new tserver with Master: 1724ebca96614b798be8a630df6afea3 (127.0.98.3:35281)
I20260929 18:58:33.660754 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45549
I20260929 18:58:33.666401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:33.689677 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:33.682576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.62:33027)
0929 18:58:33.682657 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:33.682660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:33.682750 (+ 90us) server_negotiation.cc:407] Connection header received
0929 18:58:33.682852 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:33.682853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:33.682864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:33.682934 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:33.683543 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.684032 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.684771 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:33.684966 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:33.685052 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:33.687223 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:33.687224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:33.687224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:33.687231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:33.688688 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.689032 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.689033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.689033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.689085 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:33.689243 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:33.689244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:33.689244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:33.689397 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:33.689481 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:33.689555 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 18:58:33.689612 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:58:33.689931 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43668
I20260929 18:58:33.690481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3500
I20260929 18:58:33.695225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3634
I20260929 18:58:33.699399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3768
I20260929 18:58:33.703377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3406
2026-09-29T18:58:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1834 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3909](info): set up 2 sockets
Sep 29 18:58:33 dist-test-slave-gfsd krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260929 18:58:34.739099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:34 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:34.859477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:34.940487 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:34.940734 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:34.940769 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:34.941937 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:34.942005 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:34.942031 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:34.942060 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:34.942091 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:34.943579 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:34.943851 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:34.944181 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:34.949357 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:34.949363 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:34.949532 3922 server_base.cc:1034] running on GCE node
W20260929 18:58:34.949363 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:34.949952 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:34.950279 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:34.951433 3922 hybrid_clock.cc:648] HybridClock initialized: now 1790708314951414 us; error 38 us; skew 500 ppm
Sep 29 18:58:34 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:34.954037 3922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:34.954536 3922 webserver.cc:460] Webserver started at http://127.0.98.62:46153/ using document root <none> and password file <none>
I20260929 18:58:34.954783 3922 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:34.954859 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:34.955006 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:34.956127 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7"
format_stamp: "Formatted at 2026-09-29 18:58:34 on dist-test-slave-gfsd"
server_key: "f54aec7f956cdde42891ef77705ad220"
server_key_iv: "23caac66e672285eb84547d39b7ee0a4"
server_key_version: "encryptionkey@0"
I20260929 18:58:34.956492 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7"
format_stamp: "Formatted at 2026-09-29 18:58:34 on dist-test-slave-gfsd"
server_key: "f54aec7f956cdde42891ef77705ad220"
server_key_iv: "23caac66e672285eb84547d39b7ee0a4"
server_key_version: "encryptionkey@0"
I20260929 18:58:34.957839 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:58:34.958813 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:34.958990 3922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:34.959089 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7"
format_stamp: "Formatted at 2026-09-29 18:58:34 on dist-test-slave-gfsd"
server_key: "f54aec7f956cdde42891ef77705ad220"
server_key_iv: "23caac66e672285eb84547d39b7ee0a4"
server_key_version: "encryptionkey@0"
I20260929 18:58:34.959174 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:34.966637 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:34.968622 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:34.968776 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:34.972787 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38399
I20260929 18:58:34.972818 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38399 every 8 connection(s)
I20260929 18:58:34.973170 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:34.974114 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:34.974162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 3922
I20260929 18:58:34.974268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:34.974478 392 external_mini_cluster.cc:1428] Setting key df60c655bf46f7ce02bbc55d5a70f80a
I20260929 18:58:34.977572 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7: Bootstrap starting.
I20260929 18:58:34.978333 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:34.978628 3990 log.cc:824] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:34.979393 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7: No bootstrap required, opened a new log
I20260929 18:58:34.981247 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } }
I20260929 18:58:34.981379 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:34.981401 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfaff41dbff4f42a5dc4c198e5d81f7, State: Initialized, Role: FOLLOWER
I20260929 18:58:34.981544 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } }
I20260929 18:58:34.981618 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 29 18:58:34 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:34.981667 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:34.981717 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:34.982336 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } }
I20260929 18:58:34.982455 3990 leader_election.cc:302] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dfaff41dbff4f42a5dc4c198e5d81f7; no voters:
I20260929 18:58:34.982618 3990 leader_election.cc:288] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:34.982677 3995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:34.982829 3995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6dfaff41dbff4f42a5dc4c198e5d81f7, State: Running, Role: LEADER
I20260929 18:58:34.982899 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:34.982923 3995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } }
I20260929 18:58:34.983405 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dfaff41dbff4f42a5dc4c198e5d81f7. Latest consensus state: current_term: 1 leader_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } } }
I20260929 18:58:34.983500 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:34.983735 4003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:34.984297 4003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:34.985371 3996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfaff41dbff4f42a5dc4c198e5d81f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38399 } } }
I20260929 18:58:34.985455 3996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:34.986570 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:34.975038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.98.62:38399)
0929 18:58:34.975202 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:34.975211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:34.975245 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:58:34.975798 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:34.975805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:34.975855 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:34.975968 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:34.976818 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:34.977720 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:34.978801 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:34.979393 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:34.979507 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:34.982150 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:34.982153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:34.982154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:34.982166 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:34.984672 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:34.985065 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:34.985069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:34.985069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:34.985170 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:34.985361 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:34.985361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:34.985362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:34.985656 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:34.985784 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:34.986160 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 18:58:34.986311 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260929 18:58:34.987298 4003 catalog_manager.cc:1347] Generated new cluster ID: 449c3863e72a4781bafc568ecd869ebc
I20260929 18:58:34.987393 4003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:35.002738 4003 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:35.003422 4003 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:35.013759 4003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dfaff41dbff4f42a5dc4c198e5d81f7: Generated new TSK 0
I20260929 18:58:35.014091 4003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:35.046818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:35.140859 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:35.141060 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:35.141091 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:35.142298 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:35.142336 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:35.142364 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:35.143846 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:35.144109 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:35.144340 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:35.149540 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:35.149582 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:35.149685 4016 server_base.cc:1034] running on GCE node
W20260929 18:58:35.149582 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:35.149977 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:35.150285 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:35.151441 4016 hybrid_clock.cc:648] HybridClock initialized: now 1790708315151406 us; error 51 us; skew 500 ppm
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:35.154036 4016 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:35.154421 4016 webserver.cc:460] Webserver started at http://127.0.98.1:38877/ using document root <none> and password file <none>
I20260929 18:58:35.154673 4016 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:35.154757 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:35.154902 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:35.156003 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "a31f35fefa4241ffb6e65e439bc0acde"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "6a8059a9b2697b92c0b6933a250349f0"
server_key_iv: "6c85ea18db7c0dcb2bf3cdfed6139d48"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.156368 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a31f35fefa4241ffb6e65e439bc0acde"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "6a8059a9b2697b92c0b6933a250349f0"
server_key_iv: "6c85ea18db7c0dcb2bf3cdfed6139d48"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.158021 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:35.158829 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.159032 4016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:35.159137 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "a31f35fefa4241ffb6e65e439bc0acde"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "6a8059a9b2697b92c0b6933a250349f0"
server_key_iv: "6c85ea18db7c0dcb2bf3cdfed6139d48"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.159220 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:35.168042 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:35.170387 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:35.170553 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:35.170787 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:35.171103 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:35.171157 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.171197 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:35.171226 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.177346 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33617
I20260929 18:58:35.177387 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33617 every 8 connection(s)
I20260929 18:58:35.177762 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:35.182572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4016
I20260929 18:58:35.182689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:35.182914 392 external_mini_cluster.cc:1428] Setting key 40aa7383984351b8ea9cb9100f2963da
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:35.187971 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.178465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38399 (local address 127.0.98.1:33945)
0929 18:58:35.178641 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:35.178683 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:35.179667 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:35.179904 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:35.179908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:35.180155 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:35.180338 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.180341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.181067 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.181068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.181865 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.181866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.182255 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.182256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.182355 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.182930 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:35.182938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:35.184648 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:35.186338 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.186344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.186346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.186569 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.186800 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.186801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.186802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.186915 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.187183 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:35.187189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:35.187374 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 18:58:35.187488 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:58:35.187686 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260929 18:58:35.187974 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.178488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33945 (local address 127.0.98.62:38399)
0929 18:58:35.178557 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:35.178559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:35.178775 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:58:35.179770 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:35.179771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:35.179782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:35.179848 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:35.180438 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.180975 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.181954 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.182176 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.182245 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.184749 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:35.184750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:35.184750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:35.184758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:35.186250 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.186667 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.186668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.186669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.186733 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.186984 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.186985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.186985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.187112 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:35.187192 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:35.187570 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 18:58:35.187744 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:58:35.188679 4145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38399
I20260929 18:58:35.188773 4145 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:35.188956 4145 heartbeater.cc:511] Master 127.0.98.62:38399 requested a full tablet report, sending...
I20260929 18:58:35.189504 3954 ts_manager.cc:194] Registered new tserver with Master: a31f35fefa4241ffb6e65e439bc0acde (127.0.98.1:33617)
I20260929 18:58:35.190490 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33945
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:35.210551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:35.292480 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:35.292682 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:35.292706 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:35.293855 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:35.293898 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:35.293924 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:35.295426 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:35.295661 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:35.295941 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:35.301191 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:35.301190 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:35.301338 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:35.301440 4150 server_base.cc:1034] running on GCE node
I20260929 18:58:35.301681 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:35.301935 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:35.303102 4150 hybrid_clock.cc:648] HybridClock initialized: now 1790708315303087 us; error 30 us; skew 500 ppm
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:35.305670 4150 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:35.306093 4150 webserver.cc:460] Webserver started at http://127.0.98.2:36467/ using document root <none> and password file <none>
I20260929 18:58:35.306322 4150 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:35.306388 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:35.306533 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:35.307677 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "e7892515153242d9afc676a3c1d15aa8"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "a3454abf704a3579162c50c42cb969c9"
server_key_iv: "fc820b58186ce39055d50c197c72be6f"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.308027 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e7892515153242d9afc676a3c1d15aa8"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "a3454abf704a3579162c50c42cb969c9"
server_key_iv: "fc820b58186ce39055d50c197c72be6f"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.309309 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:35.310055 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.310246 4150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:35.310350 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "e7892515153242d9afc676a3c1d15aa8"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "a3454abf704a3579162c50c42cb969c9"
server_key_iv: "fc820b58186ce39055d50c197c72be6f"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.310422 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:35.322754 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:35.324645 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:35.324805 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:35.325026 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:35.325338 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:35.325392 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.325426 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:35.325453 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.331097 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42899
I20260929 18:58:35.331146 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42899 every 8 connection(s)
I20260929 18:58:35.331494 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:35.335408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4150
I20260929 18:58:35.335520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:35.335729 392 external_mini_cluster.cc:1428] Setting key 896f60955a601f533c067aee069343e3
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:35.341439 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.332162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38399 (local address 127.0.98.2:46417)
0929 18:58:35.332326 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:35.332365 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:35.333330 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:35.333541 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:35.333544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:35.333782 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:35.333960 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.333964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.334611 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.334612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.335640 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.335643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.336122 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.336123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.336238 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.336750 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:35.336758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:35.338249 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:35.340177 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.340183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.340185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.340403 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.340622 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.340623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.340623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.340715 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.341000 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:35.341004 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:35.341105 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:58:35.341182 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:58:35.341292 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260929 18:58:35.341466 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.332210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46417 (local address 127.0.98.62:38399)
0929 18:58:35.332252 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:35.332254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:35.332452 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:58:35.333416 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:35.333417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:35.333426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:35.333481 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:35.334030 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.334532 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.335721 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.336029 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.336122 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.338342 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:35.338343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:35.338344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:35.338352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:35.340090 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.340481 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.340483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.340483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.340551 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.340782 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.340783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.340783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.340932 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:35.341018 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:35.341233 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:58:35.341326 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:58:35.341944 4279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38399
I20260929 18:58:35.342085 4279 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:35.342275 4279 heartbeater.cc:511] Master 127.0.98.62:38399 requested a full tablet report, sending...
I20260929 18:58:35.342656 3954 ts_manager.cc:194] Registered new tserver with Master: e7892515153242d9afc676a3c1d15aa8 (127.0.98.2:42899)
I20260929 18:58:35.343037 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46417
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:35.363415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:35.442404 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:35.442612 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:35.442634 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:35.443825 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:35.443867 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:35.443893 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:35.445362 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:35.445604 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:35.445843 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:35.451027 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:35.451104 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:35.451104 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:35.451122 4284 server_base.cc:1034] running on GCE node
I20260929 18:58:35.451426 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:35.451694 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:35.452883 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790708315452874 us; error 42 us; skew 500 ppm
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:35.455554 4284 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:35.455991 4284 webserver.cc:460] Webserver started at http://127.0.98.3:44389/ using document root <none> and password file <none>
I20260929 18:58:35.456233 4284 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:35.456303 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:35.456455 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:35.457631 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8b1a0a9274a412ba33eedaea06737ea"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "53e69a5bb66f939f5926cca7037cbdc1"
server_key_iv: "c5b8dda8124700018087e4b9609587dd"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.457991 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8b1a0a9274a412ba33eedaea06737ea"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "53e69a5bb66f939f5926cca7037cbdc1"
server_key_iv: "c5b8dda8124700018087e4b9609587dd"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.459703 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 18:58:35.460611 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.460819 4284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:35.460942 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "c8b1a0a9274a412ba33eedaea06737ea"
format_stamp: "Formatted at 2026-09-29 18:58:35 on dist-test-slave-gfsd"
server_key: "53e69a5bb66f939f5926cca7037cbdc1"
server_key_iv: "c5b8dda8124700018087e4b9609587dd"
server_key_version: "encryptionkey@0"
I20260929 18:58:35.461025 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:35.485723 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:35.488375 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:35.488564 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:35.488823 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:35.489152 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:35.489216 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.489264 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:35.489291 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:35.495430 4284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44007
I20260929 18:58:35.495476 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44007 every 8 connection(s)
I20260929 18:58:35.495858 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:35.498158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4284
I20260929 18:58:35.498234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:35.498407 392 external_mini_cluster.cc:1428] Setting key 79ccb0719c45b9b5730ce68d295697eb
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:35.505632 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.496667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41713 (local address 127.0.98.62:38399)
0929 18:58:35.496734 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:35.496736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:35.496979 (+ 243us) server_negotiation.cc:407] Connection header received
0929 18:58:35.497978 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:35.497979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:35.497993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:35.498082 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:35.498977 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.499660 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.500742 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.500924 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.500983 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.503186 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:35.503187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:35.503187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:35.503194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:35.504448 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.504830 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.504831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.504831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.504878 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.505140 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.505140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.505140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.505235 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:35.505284 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:35.505507 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 18:58:35.505570 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:58:35.505909 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.496658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38399 (local address 127.0.98.3:41713)
0929 18:58:35.496831 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:35.496877 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:35.497864 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:35.498137 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:35.498141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:35.498565 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:35.498884 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.498891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.499740 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.499741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.500660 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:35.500662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.501005 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.501006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:35.501109 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.501647 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:35.501656 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:35.503102 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:35.504545 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.504550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.504552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.504764 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.504934 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:35.504935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:35.504935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:35.505078 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:35.505291 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:35.505295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:35.505402 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 18:58:35.505482 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:58:35.505627 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260929 18:58:35.506506 4413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38399
I20260929 18:58:35.506611 4413 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:35.506783 4413 heartbeater.cc:511] Master 127.0.98.62:38399 requested a full tablet report, sending...
I20260929 18:58:35.507124 3954 ts_manager.cc:194] Registered new tserver with Master: c8b1a0a9274a412ba33eedaea06737ea (127.0.98.3:44007)
I20260929 18:58:35.507535 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41713
I20260929 18:58:35.509969 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:35.533886 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.525534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.98.1:33617)
0929 18:58:35.525700 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:35.525705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:35.525716 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:35.525742 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:35.525744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:35.525758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:35.525830 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:35.526469 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.527262 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.528096 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.528419 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.528483 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.530717 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:35.530724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:35.530724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:35.530731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:35.532526 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.532884 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.532887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.532887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.532948 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.533165 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.533166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.533166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.533333 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:35.533461 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:35.533634 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:58:35.533732 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:35.542205 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:35.535563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.98.62:38399)
0929 18:58:35.535626 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:35.535628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:35.535667 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:58:35.535745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:35.535746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:35.535756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:35.535811 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:35.536291 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.536775 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.537472 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:35.537669 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:35.537752 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:35.539884 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:35.539885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:35.539886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:35.539893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:35.541064 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.541399 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.541401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.541401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.541457 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:35.541652 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:35.541653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:35.541654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:35.541799 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:35.541900 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:35.542003 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:58:35.542082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:58:35.542501 3954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56852
I20260929 18:58:35.542870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4016
I20260929 18:58:35.547448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4150
I20260929 18:58:35.551931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4284
I20260929 18:58:35.556182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 3922
2026-09-29T18:58:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1853 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[4425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[4425](info): set up 2 sockets
Sep 29 18:58:35 dist-test-slave-gfsd krb5kdc[4425](info): commencing operation
krb5kdc: starting...
W20260929 18:58:36.603852 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:36 dist-test-slave-gfsd krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:36.726507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:36.806051 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:36.806254 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:36.806277 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:36.807376 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:36.807410 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:36.807423 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:36.807436 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:36.807448 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:36.808867 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:36.809053 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:36.809211 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:36.814303 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:36.814338 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:36.814376 4438 server_base.cc:1034] running on GCE node
W20260929 18:58:36.814275 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:36.814649 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:36.814966 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:36.816107 4438 hybrid_clock.cc:648] HybridClock initialized: now 1790708316816074 us; error 54 us; skew 500 ppm
Sep 29 18:58:36 dist-test-slave-gfsd krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:36.822228 4438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:36.822607 4438 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:36.822679 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:36.822819 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:36.823959 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "9418ee49f13b403897812204250deb83"
format_stamp: "Formatted at 2026-09-29 18:58:36 on dist-test-slave-gfsd"
server_key: "bd4b594a9155a3b8925868179266e59e"
server_key_iv: "2fa43bde1920767ceea173efe448e73a"
server_key_version: "encryptionkey@0"
I20260929 18:58:36.824331 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "9418ee49f13b403897812204250deb83"
format_stamp: "Formatted at 2026-09-29 18:58:36 on dist-test-slave-gfsd"
server_key: "bd4b594a9155a3b8925868179266e59e"
server_key_iv: "2fa43bde1920767ceea173efe448e73a"
server_key_version: "encryptionkey@0"
I20260929 18:58:36.825640 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:58:36.826424 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:36.826620 4438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:36.826727 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "9418ee49f13b403897812204250deb83"
format_stamp: "Formatted at 2026-09-29 18:58:36 on dist-test-slave-gfsd"
server_key: "bd4b594a9155a3b8925868179266e59e"
server_key_iv: "2fa43bde1920767ceea173efe448e73a"
server_key_version: "encryptionkey@0"
I20260929 18:58:36.826808 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:36.842849 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:36.844982 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:36.845144 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:36.849313 4438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36425
I20260929 18:58:36.849340 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36425 every 8 connection(s)
I20260929 18:58:36.849633 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:36.850529 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:36.851244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4438
I20260929 18:58:36.851377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:36.851572 392 external_mini_cluster.cc:1428] Setting key 97617360bb7f8992b872423db84ccfb4
I20260929 18:58:36.853714 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: Bootstrap starting.
I20260929 18:58:36.854302 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:36.854583 4505 log.cc:824] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:36.855261 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: No bootstrap required, opened a new log
I20260929 18:58:36.857168 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } }
I20260929 18:58:36.857319 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:36.857369 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9418ee49f13b403897812204250deb83, State: Initialized, Role: FOLLOWER
I20260929 18:58:36.857537 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } }
I20260929 18:58:36.857640 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:36.857681 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:36.857731 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:58:36 dist-test-slave-gfsd krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:36.858390 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } }
I20260929 18:58:36.858515 4505 leader_election.cc:302] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9418ee49f13b403897812204250deb83; no voters:
I20260929 18:58:36.858662 4505 leader_election.cc:288] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:36.858691 4510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:36.858789 4510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [term 1 LEADER]: Becoming Leader. State: Replica: 9418ee49f13b403897812204250deb83, State: Running, Role: LEADER
I20260929 18:58:36.858872 4510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } }
I20260929 18:58:36.858904 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:36.859197 4512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9418ee49f13b403897812204250deb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } } }
I20260929 18:58:36.859321 4512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:36.859483 4512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9418ee49f13b403897812204250deb83. Latest consensus state: current_term: 1 leader_uuid: "9418ee49f13b403897812204250deb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418ee49f13b403897812204250deb83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36425 } } }
I20260929 18:58:36.859535 4512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:36.860033 4516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:36.860885 4516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:36.863050 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:36.852073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.0.98.62:36425)
0929 18:58:36.852222 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:36.852232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:36.852258 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:58:36.852670 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:36.852677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:36.852730 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:36.852844 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:36.853596 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:36.854660 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:36.855668 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:36.856081 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:36.856201 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:36.858906 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:36.858910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:36.858910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:36.858922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:36.861381 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:36.861846 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:36.861849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:36.861849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:36.861944 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:36.862181 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:36.862182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:36.862183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:36.862492 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:36.862646 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:36.862715 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:58:36.862847 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260929 18:58:36.863844 4516 catalog_manager.cc:1347] Generated new cluster ID: 31dcdcc71f7c4fcf984443331cadf664
I20260929 18:58:36.863896 4516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:36.870982 4516 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:36.871666 4516 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:36.876030 4516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: Generated new TSK 0
I20260929 18:58:36.876233 4516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:36.910470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:36.992518 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:36.992851 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:36.992915 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:36.994639 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:36.994704 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:36.994768 4531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:36.996896 4531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:36.997154 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:36.997394 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:37.002519 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.002592 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.002516 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.002982 4531 server_base.cc:1034] running on GCE node
I20260929 18:58:37.003209 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:37.003520 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:37.004703 4531 hybrid_clock.cc:648] HybridClock initialized: now 1790708317004675 us; error 42 us; skew 500 ppm
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:37.007591 4531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:37.007898 4531 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:37.007946 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:37.008045 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:37.009202 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0c0abb5d89a4c20854842b4e9e495ff"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "dd02d4f801067f8eb3e092a8f1b62bce"
server_key_iv: "158c31fa83603449a7d658293f3a3866"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.009707 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0c0abb5d89a4c20854842b4e9e495ff"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "dd02d4f801067f8eb3e092a8f1b62bce"
server_key_iv: "158c31fa83603449a7d658293f3a3866"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.011335 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:37.012197 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.012383 4531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:37.012496 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "e0c0abb5d89a4c20854842b4e9e495ff"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "dd02d4f801067f8eb3e092a8f1b62bce"
server_key_iv: "158c31fa83603449a7d658293f3a3866"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.012619 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:37.031376 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:37.033417 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:37.033608 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:37.033872 4531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:37.034207 4531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:37.034268 4531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.034307 4531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:37.034360 4531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.040834 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44219
I20260929 18:58:37.040881 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44219 every 8 connection(s)
I20260929 18:58:37.041163 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:37.045210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4531
I20260929 18:58:37.045341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:37.045508 392 external_mini_cluster.cc:1428] Setting key f728fed22b2c55a499cab882db9c01e4
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:37.051735 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.041948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56003 (local address 127.0.98.62:36425)
0929 18:58:37.042016 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.042018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.042229 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:58:37.042723 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.042724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.042733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.042793 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:37.043811 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.044296 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.045378 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.045671 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.045771 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.048408 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:37.048410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:37.048411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:37.048420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:37.050102 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.050586 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.050588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.050589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.050658 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.050942 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.050943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.050943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.051078 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:37.051162 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.051524 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.051627 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:58:37.051963 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.041914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36425 (local address 127.0.98.1:56003)
0929 18:58:37.042087 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.042133 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.042635 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.042849 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.042853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.043156 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:37.043725 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.043731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.044384 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.044385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.045265 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.045267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.045783 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.045784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.045972 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.046600 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:37.046609 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:37.048309 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:37.050220 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.050228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.050230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.050475 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.050743 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.050744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.050744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.050860 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.051152 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:37.051160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:37.051363 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.051500 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.051701 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260929 18:58:37.052511 4659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36425
I20260929 18:58:37.052626 4659 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:37.052790 4659 heartbeater.cc:511] Master 127.0.98.62:36425 requested a full tablet report, sending...
I20260929 18:58:37.053383 4469 ts_manager.cc:194] Registered new tserver with Master: e0c0abb5d89a4c20854842b4e9e495ff (127.0.98.1:44219)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:58:37.054421 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:37.074213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:37.155958 4664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:37.156179 4664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:37.156204 4664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:37.157302 4664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:37.157334 4664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:37.157361 4664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:37.158892 4664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:37.159094 4664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:37.159376 4664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:37.164547 4670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.164618 4672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.164664 4664 server_base.cc:1034] running on GCE node
W20260929 18:58:37.164588 4669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.164922 4664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:37.165292 4664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:37.166493 4664 hybrid_clock.cc:648] HybridClock initialized: now 1790708317166473 us; error 42 us; skew 500 ppm
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:37.169382 4664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:37.169700 4664 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:37.169749 4664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:37.169843 4664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:37.170917 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "94bfdfb9abd648d7998e73b153f54569"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "28c96b230627674ab7719fd8e85a4e8c"
server_key_iv: "333b3b7828fc5cc1c65820136d2372e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.171243 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "94bfdfb9abd648d7998e73b153f54569"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "28c96b230627674ab7719fd8e85a4e8c"
server_key_iv: "333b3b7828fc5cc1c65820136d2372e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.172614 4664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:37.173467 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.173723 4664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:37.173799 4664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "94bfdfb9abd648d7998e73b153f54569"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "28c96b230627674ab7719fd8e85a4e8c"
server_key_iv: "333b3b7828fc5cc1c65820136d2372e6"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.173897 4664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:37.189435 4664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:37.191715 4664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:37.191859 4664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:37.192109 4664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:37.192420 4664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:37.192453 4664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.192476 4664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:37.192521 4664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.198307 4664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44785
I20260929 18:58:37.198385 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44785 every 8 connection(s)
I20260929 18:58:37.198650 4664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:37.199046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4664
I20260929 18:58:37.199157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:37.199371 392 external_mini_cluster.cc:1428] Setting key 02e341092c0d4d609d5bb5f2c27064a6
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:58:37.209923 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.199584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50093 (local address 127.0.98.62:36425)
0929 18:58:37.199653 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.199655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.199891 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:58:37.200546 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.200547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.200561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.200608 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:37.201707 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.202195 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.203842 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.204094 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.204160 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.206822 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:37.206824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:37.206824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:37.206834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:37.208526 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.208972 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.208973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.208973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.209031 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.209308 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.209309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.209309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.209428 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:37.209493 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.209731 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.209826 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:58:37.210116 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.199593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36425 (local address 127.0.98.2:50093)
0929 18:58:37.199752 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.199797 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.200431 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.200694 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.200703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.200948 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:37.201607 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.201614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.202286 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.202287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.203701 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.203705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.204188 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.204189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.204341 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.204847 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:37.204857 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:37.206692 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:37.208634 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.208640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.208642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.208860 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.209108 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.209109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.209109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.209221 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.209499 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:37.209503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:37.209615 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.209696 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.209861 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:58:37.210768 4792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36425
I20260929 18:58:37.210875 4792 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:37.211026 4792 heartbeater.cc:511] Master 127.0.98.62:36425 requested a full tablet report, sending...
I20260929 18:58:37.211468 4469 ts_manager.cc:194] Registered new tserver with Master: 94bfdfb9abd648d7998e73b153f54569 (127.0.98.2:44785)
I20260929 18:58:37.211874 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:37.226191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:37.309128 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:37.309356 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:37.309383 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:37.310578 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:37.310627 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:37.310657 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:37.312179 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:37.312425 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:37.312672 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:37.317942 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.318049 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.317942 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.318068 4797 server_base.cc:1034] running on GCE node
I20260929 18:58:37.318374 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:37.318652 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:37.319818 4797 hybrid_clock.cc:648] HybridClock initialized: now 1790708317319807 us; error 45 us; skew 500 ppm
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:37.322436 4797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:37.322863 4797 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:37.322948 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:37.323086 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:37.324265 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d3a33cfcb064953b5e8e55827800281"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "c7bb3021e6244a46e559fcb79f315552"
server_key_iv: "99548c13fef0acb989c76a04e6c564f7"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.324640 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d3a33cfcb064953b5e8e55827800281"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "c7bb3021e6244a46e559fcb79f315552"
server_key_iv: "99548c13fef0acb989c76a04e6c564f7"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.325934 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:37.326737 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.326921 4797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:37.327026 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "3d3a33cfcb064953b5e8e55827800281"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "c7bb3021e6244a46e559fcb79f315552"
server_key_iv: "99548c13fef0acb989c76a04e6c564f7"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.327100 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:37.350484 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:37.352944 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:37.353121 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:37.353359 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:37.353699 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:37.353754 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.353796 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:37.353823 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.359755 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46251
I20260929 18:58:37.359949 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46251 every 8 connection(s)
I20260929 18:58:37.360109 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:37.361210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4797
I20260929 18:58:37.361336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:37.361568 392 external_mini_cluster.cc:1428] Setting key ed911a0bcc0e606ccf73d69db51b7f78
Sep 29 18:58:37 dist-test-slave-gfsd krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:37.369885 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.360865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47437 (local address 127.0.98.62:36425)
0929 18:58:37.360930 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.360933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.361158 (+ 225us) server_negotiation.cc:407] Connection header received
0929 18:58:37.361815 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.361816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.361830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.361896 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:37.363239 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.363747 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.364781 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.365000 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.365110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.367252 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:37.367254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:37.367254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:37.367261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:37.368511 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.368925 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.368926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.368926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.368980 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.369286 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.369287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.369287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.369397 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:37.369458 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.369759 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.369821 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:58:37.370142 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.360835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36425 (local address 127.0.98.3:47437)
0929 18:58:37.361019 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.361058 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.361663 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.361997 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.362001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.362366 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:37.362955 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.362962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.363864 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.363865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.364692 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.364693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.365126 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.365127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.365222 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.365770 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:37.365785 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:37.367164 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:37.368600 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.368605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.368606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.368843 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.369076 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:37.369076 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:37.369077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:37.369194 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:37.369504 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:37.369510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:37.369632 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.369730 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.369850 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260929 18:58:37.370698 4925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36425
I20260929 18:58:37.370828 4925 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:37.370981 4925 heartbeater.cc:511] Master 127.0.98.62:36425 requested a full tablet report, sending...
I20260929 18:58:37.371419 4469 ts_manager.cc:194] Registered new tserver with Master: 3d3a33cfcb064953b5e8e55827800281 (127.0.98.3:46251)
I20260929 18:58:37.371801 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47437
I20260929 18:58:37.373440 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:37.380897 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.374743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.98.62:36425)
0929 18:58:37.374781 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.374783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.374822 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:58:37.374888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.374889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.374896 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.374948 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:37.375576 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.376041 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.377032 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.377225 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.377294 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.378612 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:37.378613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:37.378613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:37.378619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:37.379937 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.380280 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.380281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.380281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.380328 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:37.380494 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:37.380495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:37.380495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:37.380631 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:37.380703 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.380784 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.380840 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:37.382397 4469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:37.384150 4469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:37.390695 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46251 (local address 127.0.0.1:57826)
0929 18:58:37.386830 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.386839 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.386888 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.387150 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.387153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.387156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:37.387222 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.387224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388555 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.388556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.389544 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.389546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390372 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.390454 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390473 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.390548 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.390604 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":85,"threads_started":1}
I20260929 18:58:37.390817 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.0.98.2:44785)
0929 18:58:37.387233 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.387238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.387255 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:58:37.387326 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.387328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.387343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.387419 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:37.387643 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388339 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.389856 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390420 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390514 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390575 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.390674 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.390720 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":178,"threads_started":1}
I20260929 18:58:37.391018 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44219 (local address 127.0.0.1:32880)
0929 18:58:37.386727 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.386765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.386868 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.387443 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.387444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.387447 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:37.387508 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.387510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388739 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.388740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.389848 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.389850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390812 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.390874 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390877 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.390921 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.390955 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260929 18:58:37.391040 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.98.3:46251)
0929 18:58:37.386702 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.386708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.386884 (+ 176us) server_negotiation.cc:407] Connection header received
0929 18:58:37.386961 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.386964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.386981 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.387083 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:37.387338 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388438 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.389635 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390274 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390350 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390404 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.390910 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.390957 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260929 18:58:37.391274 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.98.1:44219)
0929 18:58:37.386796 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.386800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.386843 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:58:37.387204 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.387206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.387222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.387341 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:37.387606 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388635 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.389931 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390664 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390724 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390776 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.391139 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.391197 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260929 18:58:37.391772 4859 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb (DEFAULT_TABLE table=test-table [id=f48ad2db77c4458491d0c5c1f556ce8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:37.391913 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.386700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44785 (local address 127.0.0.1:52700)
0929 18:58:37.386974 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.386981 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.387013 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.387507 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.387507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.387509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:37.387549 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.387550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.388481 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.388482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.389745 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.389748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.390515 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.390516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.390586 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.390590 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.390650 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.390698 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":73,"threads_started":1}
I20260929 18:58:37.391928 4593 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb (DEFAULT_TABLE table=test-table [id=f48ad2db77c4458491d0c5c1f556ce8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:37.392196 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97f7ce02ac4434b93a67b48b6d59eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:37.392263 4593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97f7ce02ac4434b93a67b48b6d59eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:37.392709 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb (DEFAULT_TABLE table=test-table [id=f48ad2db77c4458491d0c5c1f556ce8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:37.392993 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97f7ce02ac4434b93a67b48b6d59eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:37.394737 4942 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Bootstrap starting.
I20260929 18:58:37.395413 4942 tablet_bootstrap.cc:654] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:37.395653 4943 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Bootstrap starting.
I20260929 18:58:37.396023 4942 log.cc:824] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:37.396277 4943 tablet_bootstrap.cc:654] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:37.396298 4944 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Bootstrap starting.
I20260929 18:58:37.396871 4943 log.cc:824] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:37.396951 4944 tablet_bootstrap.cc:654] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:37.397498 4944 log.cc:824] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:37.397691 4943 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: No bootstrap required, opened a new log
I20260929 18:58:37.397768 4943 ts_tablet_manager.cc:1397] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:37.398223 4942 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: No bootstrap required, opened a new log
I20260929 18:58:37.398273 4944 tablet_bootstrap.cc:492] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: No bootstrap required, opened a new log
I20260929 18:58:37.398313 4942 ts_tablet_manager.cc:1397] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 18:58:37.398329 4944 ts_tablet_manager.cc:1397] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:37.399835 4943 raft_consensus.cc:348] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400008 4943 raft_consensus.cc:374] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:37.400051 4943 raft_consensus.cc:729] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c0abb5d89a4c20854842b4e9e495ff, State: Initialized, Role: FOLLOWER
I20260929 18:58:37.399991 4942 raft_consensus.cc:348] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400221 4942 raft_consensus.cc:374] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:37.400202 4943 consensus_queue.cc:260] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400274 4942 raft_consensus.cc:729] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3a33cfcb064953b5e8e55827800281, State: Initialized, Role: FOLLOWER
I20260929 18:58:37.400303 4944 raft_consensus.cc:348] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400405 4942 consensus_queue.cc:260] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400460 4944 raft_consensus.cc:374] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:37.400496 4944 raft_consensus.cc:729] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94bfdfb9abd648d7998e73b153f54569, State: Initialized, Role: FOLLOWER
I20260929 18:58:37.400499 4943 ts_tablet_manager.cc:1428] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:37.400646 4659 heartbeater.cc:503] Master 127.0.98.62:36425 was elected leader, sending a full tablet report...
I20260929 18:58:37.400636 4944 consensus_queue.cc:260] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.400709 4942 ts_tablet_manager.cc:1428] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 18:58:37.400825 4925 heartbeater.cc:503] Master 127.0.98.62:36425 was elected leader, sending a full tablet report...
I20260929 18:58:37.401189 4944 ts_tablet_manager.cc:1428] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:37.401221 4792 heartbeater.cc:503] Master 127.0.98.62:36425 was elected leader, sending a full tablet report...
W20260929 18:58:37.449270 4793 tablet.cc:2369] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:58:37.541926 4660 tablet.cc:2369] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:58:37.610656 4926 tablet.cc:2369] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:37.621970 4950 raft_consensus.cc:482] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:37.622094 4950 raft_consensus.cc:504] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.622411 4950 leader_election.cc:288] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3a33cfcb064953b5e8e55827800281 (127.0.98.3:46251), e0c0abb5d89a4c20854842b4e9e495ff (127.0.98.1:44219)
I20260929 18:58:37.625490 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.622819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51061 (local address 127.0.98.1:44219)
0929 18:58:37.622890 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.622892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.622904 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:58:37.622933 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.622935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.622948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.623078 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:37.623249 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.623771 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.624786 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.625148 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.625202 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.625211 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.625377 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.625422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:58:37.625509 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.622614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44219 (local address 127.0.98.2:51061)
0929 18:58:37.622671 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.622685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.622734 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.623095 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.623096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.623098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:37.623160 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.623162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.623881 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.623881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.624686 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.624688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.625230 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.625231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.625303 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.625313 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.625355 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.625387 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":31}
I20260929 18:58:37.625546 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.622782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53199 (local address 127.0.98.3:46251)
0929 18:58:37.622838 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.622840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.622908 (+ 68us) server_negotiation.cc:407] Connection header received
0929 18:58:37.622977 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.622978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.622988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.623064 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:37.623217 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.623740 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.624782 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.625164 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.625251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.625260 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.625441 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.625477 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 18:58:37.625656 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.622614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46251 (local address 127.0.98.2:53199)
0929 18:58:37.622810 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.622822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:37.622912 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:37.623059 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:37.623061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:37.623063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:37.623151 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.623153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.623825 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.623826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.624686 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:37.624688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.625269 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.625270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:37.625362 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.625367 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:37.625423 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:58:37.625459 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":51,"thread_start_us":120,"threads_started":1}
I20260929 18:58:37.625941 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97f7ce02ac4434b93a67b48b6d59eb" candidate_uuid: "94bfdfb9abd648d7998e73b153f54569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" is_pre_election: true
I20260929 18:58:37.625995 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97f7ce02ac4434b93a67b48b6d59eb" candidate_uuid: "94bfdfb9abd648d7998e73b153f54569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3a33cfcb064953b5e8e55827800281" is_pre_election: true
I20260929 18:58:37.626102 4613 raft_consensus.cc:2454] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94bfdfb9abd648d7998e73b153f54569 in term 0.
I20260929 18:58:37.626165 4879 raft_consensus.cc:2454] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94bfdfb9abd648d7998e73b153f54569 in term 0.
I20260929 18:58:37.626349 4682 leader_election.cc:302] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94bfdfb9abd648d7998e73b153f54569, e0c0abb5d89a4c20854842b4e9e495ff; no voters:
I20260929 18:58:37.626597 4950 raft_consensus.cc:2781] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:37.626672 4950 raft_consensus.cc:482] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:37.626701 4950 raft_consensus.cc:3037] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:37.627533 4950 raft_consensus.cc:504] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.627691 4950 leader_election.cc:288] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3a33cfcb064953b5e8e55827800281 (127.0.98.3:46251), e0c0abb5d89a4c20854842b4e9e495ff (127.0.98.1:44219)
I20260929 18:58:37.627923 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97f7ce02ac4434b93a67b48b6d59eb" candidate_uuid: "94bfdfb9abd648d7998e73b153f54569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c0abb5d89a4c20854842b4e9e495ff"
I20260929 18:58:37.628008 4613 raft_consensus.cc:3037] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:37.628002 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97f7ce02ac4434b93a67b48b6d59eb" candidate_uuid: "94bfdfb9abd648d7998e73b153f54569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3a33cfcb064953b5e8e55827800281"
I20260929 18:58:37.628094 4879 raft_consensus.cc:3037] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:37.628706 4613 raft_consensus.cc:2454] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bfdfb9abd648d7998e73b153f54569 in term 1.
I20260929 18:58:37.628891 4682 leader_election.cc:302] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94bfdfb9abd648d7998e73b153f54569, e0c0abb5d89a4c20854842b4e9e495ff; no voters:
I20260929 18:58:37.628894 4879 raft_consensus.cc:2454] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94bfdfb9abd648d7998e73b153f54569 in term 1.
I20260929 18:58:37.628981 4950 raft_consensus.cc:2781] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:37.629134 4950 raft_consensus.cc:686] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 1 LEADER]: Becoming Leader. State: Replica: 94bfdfb9abd648d7998e73b153f54569, State: Running, Role: LEADER
I20260929 18:58:37.629236 4950 consensus_queue.cc:237] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } }
I20260929 18:58:37.630134 4468 catalog_manager.cc:5495] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94bfdfb9abd648d7998e73b153f54569 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "94bfdfb9abd648d7998e73b153f54569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94bfdfb9abd648d7998e73b153f54569" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:37.675577 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.672700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.98.2:44785)
0929 18:58:37.672769 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.672771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.672779 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:37.672821 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.672822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.672840 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.672919 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:37.673071 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.673609 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.674408 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:37.674647 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:37.674746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:37.675167 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:37.675325 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:37.675408 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.675447 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.675468 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:58:37.677589 4613 raft_consensus.cc:1261] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff [term 1 FOLLOWER]: Refusing update from remote peer 94bfdfb9abd648d7998e73b153f54569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:37.677599 4879 raft_consensus.cc:1261] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 1 FOLLOWER]: Refusing update from remote peer 94bfdfb9abd648d7998e73b153f54569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:37.678011 4952 consensus_queue.cc:1035] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3a33cfcb064953b5e8e55827800281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:37.678223 4950 consensus_queue.cc:1035] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c0abb5d89a4c20854842b4e9e495ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:37.681381 4955 mvcc.cc:205] Tried to move back new op lower bound from 7334741269204209664 to 7334741269009813504. Current Snapshot: MvccSnapshot[applied={T|T < 7334741269204209664}]
I20260929 18:58:37.683501 4469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:37.683604 4469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:37.684459 4469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9418ee49f13b403897812204250deb83: Sending DeleteTablet for 3 replicas of tablet 1b97f7ce02ac4434b93a67b48b6d59eb
I20260929 18:58:37.684959 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52700
I20260929 18:58:37.684904 4859 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57826
I20260929 18:58:37.685232 4965 tablet_replica.cc:330] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: stopping tablet replica
I20260929 18:58:37.685242 4593 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b97f7ce02ac4434b93a67b48b6d59eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32880
I20260929 18:58:37.685343 4965 raft_consensus.cc:2229] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:37.685487 4965 raft_consensus.cc:2258] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:37.685490 4967 tablet_replica.cc:330] T 1b97f7ce02ac4434b93a67b48b6d59eb P e0c0abb5d89a4c20854842b4e9e495ff: stopping tablet replica
I20260929 18:58:37.685671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4531
I20260929 18:58:37.686388 4965 ts_tablet_manager.cc:1905] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:37.688549 4965 ts_tablet_manager.cc:1918] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:37.688630 4965 log.cc:1196] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/1b97f7ce02ac4434b93a67b48b6d59eb
I20260929 18:58:37.688917 4965 ts_tablet_manager.cc:1939] T 1b97f7ce02ac4434b93a67b48b6d59eb P 94bfdfb9abd648d7998e73b153f54569: Deleting consensus metadata
I20260929 18:58:37.689496 4453 catalog_manager.cc:4841] TS 94bfdfb9abd648d7998e73b153f54569 (127.0.98.2:44785): tablet 1b97f7ce02ac4434b93a67b48b6d59eb (table test-table [id=f48ad2db77c4458491d0c5c1f556ce8c]) successfully deleted
W20260929 18:58:37.692679 4456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44219 (error 108)
I20260929 18:58:37.692862 4966 tablet_replica.cc:330] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: stopping tablet replica
I20260929 18:58:37.692862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4664
I20260929 18:58:37.693334 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.693209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44219 (local address 127.0.0.1:32884)
0929 18:58:37.693259 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:37.693291 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 18:58:37.693475 4456 catalog_manager.cc:4584] TS e0c0abb5d89a4c20854842b4e9e495ff (127.0.98.1:44219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b97f7ce02ac4434b93a67b48b6d59eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44219: connect: Connection refused (error 111)
I20260929 18:58:37.693507 4966 raft_consensus.cc:2229] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:37.693670 4966 raft_consensus.cc:2258] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:37.694252 4966 ts_tablet_manager.cc:1905] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:37.696007 4966 ts_tablet_manager.cc:1918] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:37.696071 4966 log.cc:1196] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/1b97f7ce02ac4434b93a67b48b6d59eb
I20260929 18:58:37.696357 4966 ts_tablet_manager.cc:1939] T 1b97f7ce02ac4434b93a67b48b6d59eb P 3d3a33cfcb064953b5e8e55827800281: Deleting consensus metadata
I20260929 18:58:37.696794 4453 catalog_manager.cc:4841] TS 3d3a33cfcb064953b5e8e55827800281 (127.0.98.3:46251): tablet 1b97f7ce02ac4434b93a67b48b6d59eb (table test-table [id=f48ad2db77c4458491d0c5c1f556ce8c]) successfully deleted
I20260929 18:58:37.698328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4797
I20260929 18:58:37.703223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4438
2026-09-29T18:58:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2147 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:37Z Disabled control of system clock
I20260929 18:58:37.730337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37559
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:58:37.810575 4972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:37.810776 4972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:37.810796 4972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:37.811972 4972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:37.812011 4972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:37.812023 4972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:37.812036 4972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:37.812047 4972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:37.813391 4972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37559
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:37.813640 4972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:37.813943 4972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:37.818854 4978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.818897 4977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.819022 4972 server_base.cc:1034] running on GCE node
W20260929 18:58:37.818886 4980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.819398 4972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:37.819651 4972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:37.820788 4972 hybrid_clock.cc:648] HybridClock initialized: now 1790708317820777 us; error 31 us; skew 500 ppm
I20260929 18:58:37.821972 4972 webserver.cc:460] Webserver started at http://127.0.98.62:42027/ using document root <none> and password file <none>
I20260929 18:58:37.822242 4972 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:37.822315 4972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:37.822458 4972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:37.823551 4972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "23d6d24987df40b8b8944bc3da2b8028"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "6105b54fccca8a145dee781d45f3189c"
server_key_iv: "53966ef8101eb50a25f32566a1a00f67"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.823895 4972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "23d6d24987df40b8b8944bc3da2b8028"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "6105b54fccca8a145dee781d45f3189c"
server_key_iv: "53966ef8101eb50a25f32566a1a00f67"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.825245 4972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:37.825989 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.826203 4972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:37.826303 4972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "23d6d24987df40b8b8944bc3da2b8028"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "6105b54fccca8a145dee781d45f3189c"
server_key_iv: "53966ef8101eb50a25f32566a1a00f67"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.826382 4972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:37.829668 4972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:37.829962 4972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:37.830101 4972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:37.833797 4972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37559
I20260929 18:58:37.833824 5038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37559 every 8 connection(s)
I20260929 18:58:37.834147 4972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:37.834414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 4972
I20260929 18:58:37.834517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:37.834705 392 external_mini_cluster.cc:1428] Setting key 4b2f9f65e6e0a03e77c452376fd932b6
I20260929 18:58:37.834924 5039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:37.837687 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:37.835582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.98.62:37559)
0929 18:58:37.835766 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:37.835773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:37.835796 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:58:37.836387 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:37.836393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:37.836430 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:37.836576 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:37.837278 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:37.837282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:37.837282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:37.837286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:37.837329 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:37.837339 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:37.837414 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:37.837482 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:58:37.837537 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260929 18:58:37.838053 5039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: Bootstrap starting.
I20260929 18:58:37.838706 5039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:37.838973 5039 log.cc:824] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:37.839622 5039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: No bootstrap required, opened a new log
I20260929 18:58:37.840935 5039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 18:58:37.841084 5039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:37.841131 5039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d6d24987df40b8b8944bc3da2b8028, State: Initialized, Role: FOLLOWER
I20260929 18:58:37.841255 5039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 18:58:37.841344 5039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:37.841387 5039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:37.841435 5039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:37.841934 5039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 18:58:37.842051 5039 leader_election.cc:302] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d6d24987df40b8b8944bc3da2b8028; no voters:
I20260929 18:58:37.842218 5039 leader_election.cc:288] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:37.842451 5044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:37.842469 5039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:37.842613 5044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [term 1 LEADER]: Becoming Leader. State: Replica: 23d6d24987df40b8b8944bc3da2b8028, State: Running, Role: LEADER
I20260929 18:58:37.842720 5044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 18:58:37.843075 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d6d24987df40b8b8944bc3da2b8028. Latest consensus state: current_term: 1 leader_uuid: "23d6d24987df40b8b8944bc3da2b8028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } } }
I20260929 18:58:37.843176 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:37.843084 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d6d24987df40b8b8944bc3da2b8028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d6d24987df40b8b8944bc3da2b8028" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } } }
I20260929 18:58:37.843215 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:37.843396 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:37.843866 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:37.845438 5053 catalog_manager.cc:1347] Generated new cluster ID: dc00eea350524cb2a3c676865fb99caa
I20260929 18:58:37.845490 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:37.865933 5053 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:37.866115 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:37.882362 5053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: Generated new TSK 0
I20260929 18:58:37.882588 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:58:37.886359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--builtin_ntp_servers=127.0.98.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:58:37.980211 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:37.980450 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:37.980489 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:37.981643 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:37.981704 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:37.981747 5063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:37.983101 5063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:37.983402 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:37.983712 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:58:37.988904 5063 server_base.cc:1034] running on GCE node
W20260929 18:58:37.989029 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.988888 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:37.988859 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:37.989380 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:37.989624 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:37.990800 5063 hybrid_clock.cc:648] HybridClock initialized: now 1790708317990785 us; error 31 us; skew 500 ppm
I20260929 18:58:37.992308 5063 webserver.cc:460] Webserver started at http://127.0.98.1:35869/ using document root <none> and password file <none>
I20260929 18:58:37.992580 5063 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:37.992650 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:37.992795 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:37.993906 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "55dc6aa61c694d40a7757b80f4ee727f"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "95f8d4fd5e3ae593836ace54b65ba40c"
server_key_iv: "0f5d967c420c160b9ce85f939a6bae64"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.994277 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55dc6aa61c694d40a7757b80f4ee727f"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "95f8d4fd5e3ae593836ace54b65ba40c"
server_key_iv: "0f5d967c420c160b9ce85f939a6bae64"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.995730 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:37.996438 5077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:37.996595 5063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:37.996685 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "55dc6aa61c694d40a7757b80f4ee727f"
format_stamp: "Formatted at 2026-09-29 18:58:37 on dist-test-slave-gfsd"
server_key: "95f8d4fd5e3ae593836ace54b65ba40c"
server_key_iv: "0f5d967c420c160b9ce85f939a6bae64"
server_key_version: "encryptionkey@0"
I20260929 18:58:37.996740 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:38.000084 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:38.000311 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:38.000407 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:38.000552 5063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:38.000854 5063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:38.000885 5063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.000905 5063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:38.000919 5063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.007038 5063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44953
I20260929 18:58:38.007117 5190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44953 every 8 connection(s)
I20260929 18:58:38.007476 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:38.010331 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.008234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43629 (local address 127.0.98.62:37559)
0929 18:58:38.008313 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.008316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.008502 (+ 186us) server_negotiation.cc:407] Connection header received
0929 18:58:38.008965 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.008967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.008971 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.009026 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.010134 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.010135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.010136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.010139 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.010154 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.010159 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.010205 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.010266 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.010278 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:58:38.010512 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.008202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.1:43629)
0929 18:58:38.008379 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.008414 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.008882 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.009078 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.009081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.009089 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.009978 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.009978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.010012 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.010041 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.010205 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.010207 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.010210 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.010262 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.010310 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260929 18:58:38.011000 5191 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 18:58:38.011090 5191 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:38.011207 5191 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
I20260929 18:58:38.011765 5003 ts_manager.cc:194] Registered new tserver with Master: 55dc6aa61c694d40a7757b80f4ee727f (127.0.98.1:44953)
I20260929 18:58:38.011906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5063
I20260929 18:58:38.012017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:38.012208 392 external_mini_cluster.cc:1428] Setting key bfd2fed77410cfb9a940e47e9c718e26
I20260929 18:58:38.013067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--builtin_ntp_servers=127.0.98.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:58:38.094103 5194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:38.094309 5194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:38.094331 5194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:38.095554 5194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:38.095597 5194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:38.095623 5194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:38.096973 5194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:38.097218 5194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:38.097544 5194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:38.103057 5200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.103077 5199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.103150 5202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:38.103381 5194 server_base.cc:1034] running on GCE node
I20260929 18:58:38.103560 5194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:38.103745 5194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:38.104903 5194 hybrid_clock.cc:648] HybridClock initialized: now 1790708318104889 us; error 30 us; skew 500 ppm
I20260929 18:58:38.106389 5194 webserver.cc:460] Webserver started at http://127.0.98.2:40591/ using document root <none> and password file <none>
I20260929 18:58:38.106681 5194 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:38.106760 5194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:38.106941 5194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:38.108196 5194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca1583a9f7514971be9ad73bc060b736"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "7d06368d4cd8f4fa785e7d5c552f3fd0"
server_key_iv: "40a7a9b0490ec40c2ac5b2a7a25d5cfe"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.108595 5194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca1583a9f7514971be9ad73bc060b736"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "7d06368d4cd8f4fa785e7d5c552f3fd0"
server_key_iv: "40a7a9b0490ec40c2ac5b2a7a25d5cfe"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.110096 5194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:58:38.110913 5208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.111136 5194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:38.111251 5194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "ca1583a9f7514971be9ad73bc060b736"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "7d06368d4cd8f4fa785e7d5c552f3fd0"
server_key_iv: "40a7a9b0490ec40c2ac5b2a7a25d5cfe"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.111387 5194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:38.114957 5194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:38.115269 5194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:38.115480 5194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:38.115713 5194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:38.116070 5194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:38.116142 5194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.116184 5194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:38.116214 5194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.122362 5194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36471
I20260929 18:58:38.122437 5321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36471 every 8 connection(s)
I20260929 18:58:38.122792 5194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:38.126058 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.123643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52345 (local address 127.0.98.62:37559)
0929 18:58:38.123715 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.123717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.123959 (+ 242us) server_negotiation.cc:407] Connection header received
0929 18:58:38.124510 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.124511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.124518 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.124582 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.125846 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.125847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.125847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.125851 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.125866 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.125870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.125924 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.125989 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.126002 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:58:38.126344 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.123630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.2:52345)
0929 18:58:38.123813 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.123856 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.124413 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.124639 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.124642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.124651 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.125694 (+ 1043us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.125694 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.125724 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.125754 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.125918 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.125921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.125923 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.125979 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.126028 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260929 18:58:38.126960 5322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 18:58:38.127089 5322 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:38.127205 5322 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
I20260929 18:58:38.127660 5003 ts_manager.cc:194] Registered new tserver with Master: ca1583a9f7514971be9ad73bc060b736 (127.0.98.2:36471)
I20260929 18:58:38.127796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5194
I20260929 18:58:38.127914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:38.128149 392 external_mini_cluster.cc:1428] Setting key 572c1ca766f2ded0527457767f0515fa
I20260929 18:58:38.128984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--builtin_ntp_servers=127.0.98.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:58:38.213043 5325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:38.213295 5325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:38.213337 5325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:38.214620 5325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:38.214687 5325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:38.214756 5325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:38.216368 5325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:38.216653 5325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:38.216958 5325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:38.222615 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.222609 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.222608 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:38.222813 5325 server_base.cc:1034] running on GCE node
I20260929 18:58:38.223101 5325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:38.223409 5325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:38.224583 5325 hybrid_clock.cc:648] HybridClock initialized: now 1790708318224556 us; error 40 us; skew 500 ppm
I20260929 18:58:38.226109 5325 webserver.cc:460] Webserver started at http://127.0.98.3:39771/ using document root <none> and password file <none>
I20260929 18:58:38.226379 5325 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:38.226432 5325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:38.226616 5325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:38.227818 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "b51615abda2d4999ac6805bac2da240a"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "396ec1b4d278b8d1b110c0cd79412c7f"
server_key_iv: "03c326a6562f13c49b5efa2a081c83b9"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.228200 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b51615abda2d4999ac6805bac2da240a"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "396ec1b4d278b8d1b110c0cd79412c7f"
server_key_iv: "03c326a6562f13c49b5efa2a081c83b9"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.229702 5325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:38.230461 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.230666 5325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:38.230772 5325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "b51615abda2d4999ac6805bac2da240a"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "396ec1b4d278b8d1b110c0cd79412c7f"
server_key_iv: "03c326a6562f13c49b5efa2a081c83b9"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.230854 5325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:38.234407 5325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:38.234686 5325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:38.234839 5325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:38.235056 5325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:38.235431 5325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:38.235491 5325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.235533 5325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:38.235563 5325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.241909 5325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34173
I20260929 18:58:38.241971 5452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34173 every 8 connection(s)
I20260929 18:58:38.242342 5325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:38.244102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5325
I20260929 18:58:38.244211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:38.244396 392 external_mini_cluster.cc:1428] Setting key 1344eb9ef85292fb9b3aeae7536b0655
I20260929 18:58:38.245704 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.243252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35749 (local address 127.0.98.62:37559)
0929 18:58:38.243331 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.243333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.243550 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:58:38.244121 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.244123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.244130 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.244218 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.245501 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.245502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.245502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.245506 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.245522 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.245527 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.245570 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.245643 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.245654 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:58:38.245918 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.243203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.3:35749)
0929 18:58:38.243423 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.243464 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.243947 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.244308 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.244312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.244322 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.245359 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.245360 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.245388 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.245417 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.245579 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.245581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.245583 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.245639 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.245683 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260929 18:58:38.246435 5453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 18:58:38.246533 5453 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:38.246685 5453 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
I20260929 18:58:38.247076 5003 ts_manager.cc:194] Registered new tserver with Master: b51615abda2d4999ac6805bac2da240a (127.0.98.3:34173)
I20260929 18:58:38.256006 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:38.258646 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.257348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.62:37559)
0929 18:58:38.257427 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.257430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.257560 (+ 130us) server_negotiation.cc:407] Connection header received
0929 18:58:38.257646 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.257648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.257653 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.257734 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.258404 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.258405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.258405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.258410 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.258425 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.258429 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.258486 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.258553 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.258569 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:58:38.260423 5003 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:38.262099 5003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:38.266281 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44953 (local address 127.0.0.1:49236)
0929 18:58:38.264544 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.264593 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.264647 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.265050 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.265057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.265064 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.265665 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.265665 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.265685 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.265711 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.266102 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.266108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.266112 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.266199 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.266214 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260929 18:58:38.266302 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.98.1:44953)
0929 18:58:38.264894 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.264900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.264910 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:38.264936 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.264938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.264948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.265033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.265983 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.265987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.265987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.265990 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.266020 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.266032 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.266107 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.266210 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.266226 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":93,"threads_started":1}
I20260929 18:58:38.266462 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.0.1:34016)
0929 18:58:38.264725 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.264737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.264788 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.265538 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.265539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.265541 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.266039 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.266039 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.266046 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.266059 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.266321 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.266322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.266323 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.266359 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.266373 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":70,"threads_started":1}
I20260929 18:58:38.266605 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36471 (local address 127.0.0.1:47602)
0929 18:58:38.264728 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.264735 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.264777 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.265224 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.265225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.265228 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.265819 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.265819 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.265826 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.265838 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.266424 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.266425 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.266425 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.266466 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.266479 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260929 18:58:38.266654 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.0.98.2:36471)
0929 18:58:38.265056 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.265061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.265071 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:38.265105 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.265107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.265117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.265211 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.266286 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.266290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.266291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.266295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.266325 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.266338 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.266525 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.266549 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.266566 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260929 18:58:38.266918 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.264462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.0.98.3:34173)
0929 18:58:38.265413 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.265417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.265426 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:38.265451 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.265453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.265461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.265539 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.266161 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.266168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.266169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.266174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.266211 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.266233 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.266786 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.266811 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.266828 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":98,"threads_started":1}
I20260929 18:58:38.267066 5125 tablet_service.cc:1467] Processing CreateTablet for tablet 319f136312b249449ee48241866cda58 (DEFAULT_TABLE table=test-table [id=d23798eb03ac4fe1950427d4aa09079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:38.267493 5125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f136312b249449ee48241866cda58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:38.267236 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 319f136312b249449ee48241866cda58 (DEFAULT_TABLE table=test-table [id=d23798eb03ac4fe1950427d4aa09079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:38.267843 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f136312b249449ee48241866cda58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:38.267838 5387 tablet_service.cc:1467] Processing CreateTablet for tablet 319f136312b249449ee48241866cda58 (DEFAULT_TABLE table=test-table [id=d23798eb03ac4fe1950427d4aa09079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:38.268134 5387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319f136312b249449ee48241866cda58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:38.270227 5467 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Bootstrap starting.
I20260929 18:58:38.270838 5467 tablet_bootstrap.cc:654] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:38.271164 5467 log.cc:824] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:38.271319 5468 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Bootstrap starting.
I20260929 18:58:38.271593 5469 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Bootstrap starting.
I20260929 18:58:38.271911 5468 tablet_bootstrap.cc:654] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:38.272003 5467 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: No bootstrap required, opened a new log
I20260929 18:58:38.272075 5467 ts_tablet_manager.cc:1397] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:38.272228 5469 tablet_bootstrap.cc:654] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:38.272240 5468 log.cc:824] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:38.272529 5469 log.cc:824] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:38.273023 5468 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: No bootstrap required, opened a new log
I20260929 18:58:38.273103 5468 ts_tablet_manager.cc:1397] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:38.274186 5467 raft_consensus.cc:348] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.274317 5467 raft_consensus.cc:374] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:38.274338 5467 raft_consensus.cc:729] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55dc6aa61c694d40a7757b80f4ee727f, State: Initialized, Role: FOLLOWER
I20260929 18:58:38.274462 5467 consensus_queue.cc:260] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.274739 5467 ts_tablet_manager.cc:1428] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:38.274855 5191 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
I20260929 18:58:38.275180 5468 raft_consensus.cc:348] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.275399 5468 raft_consensus.cc:374] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:38.275413 5469 tablet_bootstrap.cc:492] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: No bootstrap required, opened a new log
I20260929 18:58:38.275446 5468 raft_consensus.cc:729] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1583a9f7514971be9ad73bc060b736, State: Initialized, Role: FOLLOWER
I20260929 18:58:38.275470 5469 ts_tablet_manager.cc:1397] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:58:38.275584 5468 consensus_queue.cc:260] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.275828 5468 ts_tablet_manager.cc:1428] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:38.275874 5322 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
I20260929 18:58:38.278185 5469 raft_consensus.cc:348] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.278306 5469 raft_consensus.cc:374] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:38.278326 5469 raft_consensus.cc:729] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51615abda2d4999ac6805bac2da240a, State: Initialized, Role: FOLLOWER
I20260929 18:58:38.278425 5469 consensus_queue.cc:260] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.278627 5469 ts_tablet_manager.cc:1428] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:38.278703 5453 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
W20260929 18:58:38.373442 5323 tablet.cc:2369] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:38.477396 5474 raft_consensus.cc:482] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:38.477528 5474 raft_consensus.cc:504] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.478138 5474 leader_election.cc:288] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b51615abda2d4999ac6805bac2da240a (127.0.98.3:34173), 55dc6aa61c694d40a7757b80f4ee727f (127.0.98.1:44953)
I20260929 18:58:38.479689 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.478337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.98.2:40767)
0929 18:58:38.478395 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.478407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.478449 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.478650 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.478650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.478653 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.479269 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.479270 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.479330 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.479348 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.479559 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.479560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.479561 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.479617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.479632 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:58:38.479738 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.478427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40767 (local address 127.0.98.3:34173)
0929 18:58:38.478509 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.478512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.478531 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:58:38.478561 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.478562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.478568 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.478631 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.479448 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.479450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.479450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.479456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.479475 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.479481 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.479544 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.479628 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.479644 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:58:38.479909 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.478437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59843 (local address 127.0.98.1:44953)
0929 18:58:38.478495 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.478498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.478795 (+ 297us) server_negotiation.cc:407] Connection header received
0929 18:58:38.478819 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.478820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.478825 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.478872 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.479644 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.479645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.479645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.479650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.479667 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.479673 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.479737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.479830 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.479846 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:58:38.480082 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.478378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44953 (local address 127.0.98.2:59843)
0929 18:58:38.478553 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.478561 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.478614 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.478961 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.478962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.478965 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:38.479539 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:58:38.479539 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:38.479546 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:58:38.479559 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:38.479744 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:38.479745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:38.479746 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:58:38.479997 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 18:58:38.480012 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260929 18:58:38.480221 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f136312b249449ee48241866cda58" candidate_uuid: "ca1583a9f7514971be9ad73bc060b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51615abda2d4999ac6805bac2da240a" is_pre_election: true
I20260929 18:58:38.480391 5407 raft_consensus.cc:2454] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1583a9f7514971be9ad73bc060b736 in term 0.
I20260929 18:58:38.480428 5145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f136312b249449ee48241866cda58" candidate_uuid: "ca1583a9f7514971be9ad73bc060b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55dc6aa61c694d40a7757b80f4ee727f" is_pre_election: true
I20260929 18:58:38.480615 5145 raft_consensus.cc:2454] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1583a9f7514971be9ad73bc060b736 in term 0.
I20260929 18:58:38.480702 5210 leader_election.cc:302] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b51615abda2d4999ac6805bac2da240a, ca1583a9f7514971be9ad73bc060b736; no voters:
I20260929 18:58:38.480834 5474 raft_consensus.cc:2781] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:38.480904 5474 raft_consensus.cc:482] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:38.480944 5474 raft_consensus.cc:3037] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:38.481673 5474 raft_consensus.cc:504] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.481818 5474 leader_election.cc:288] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [CANDIDATE]: Term 1 election: Requested vote from peers b51615abda2d4999ac6805bac2da240a (127.0.98.3:34173), 55dc6aa61c694d40a7757b80f4ee727f (127.0.98.1:44953)
I20260929 18:58:38.481945 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f136312b249449ee48241866cda58" candidate_uuid: "ca1583a9f7514971be9ad73bc060b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51615abda2d4999ac6805bac2da240a"
I20260929 18:58:38.481963 5145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319f136312b249449ee48241866cda58" candidate_uuid: "ca1583a9f7514971be9ad73bc060b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55dc6aa61c694d40a7757b80f4ee727f"
I20260929 18:58:38.482024 5407 raft_consensus.cc:3037] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:38.482040 5145 raft_consensus.cc:3037] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:38.482903 5407 raft_consensus.cc:2454] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1583a9f7514971be9ad73bc060b736 in term 1.
I20260929 18:58:38.482903 5145 raft_consensus.cc:2454] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1583a9f7514971be9ad73bc060b736 in term 1.
I20260929 18:58:38.483151 5209 leader_election.cc:302] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55dc6aa61c694d40a7757b80f4ee727f, ca1583a9f7514971be9ad73bc060b736; no voters:
I20260929 18:58:38.483255 5474 raft_consensus.cc:2781] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:38.483513 5474 raft_consensus.cc:686] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 1 LEADER]: Becoming Leader. State: Replica: ca1583a9f7514971be9ad73bc060b736, State: Running, Role: LEADER
I20260929 18:58:38.483634 5474 consensus_queue.cc:237] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } }
I20260929 18:58:38.484457 5003 catalog_manager.cc:5495] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1583a9f7514971be9ad73bc060b736 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ca1583a9f7514971be9ad73bc060b736" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1583a9f7514971be9ad73bc060b736" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36471 } health_report { overall_health: HEALTHY } } }
W20260929 18:58:38.492924 5454 tablet.cc:2369] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:38.493582 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.492546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.0.98.2:36471)
0929 18:58:38.492626 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.492628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.492636 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:38.492665 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.492666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.492676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.492732 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.493364 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.493365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.493366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.493370 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.493385 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.493389 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.493453 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.493516 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.493528 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:58:38.495517 5145 raft_consensus.cc:1261] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 1 FOLLOWER]: Refusing update from remote peer ca1583a9f7514971be9ad73bc060b736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:38.495517 5407 raft_consensus.cc:1261] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 1 FOLLOWER]: Refusing update from remote peer ca1583a9f7514971be9ad73bc060b736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:38.495901 5474 consensus_queue.cc:1035] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55dc6aa61c694d40a7757b80f4ee727f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:38.496074 5477 consensus_queue.cc:1035] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51615abda2d4999ac6805bac2da240a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:38.498459 5480 mvcc.cc:205] Tried to move back new op lower bound from 7334741272554397696 to 7334741272509628416. Current Snapshot: MvccSnapshot[applied={T|T < 7334741272554397696}]
I20260929 18:58:38.498708 5482 mvcc.cc:205] Tried to move back new op lower bound from 7334741272554397696 to 7334741272509628416. Current Snapshot: MvccSnapshot[applied={T|T < 7334741272554397696}]
I20260929 18:58:38.499053 5481 mvcc.cc:205] Tried to move back new op lower bound from 7334741272554397696 to 7334741272509628416. Current Snapshot: MvccSnapshot[applied={T|T < 7334741272554397696}]
I20260929 18:58:38.501303 5003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:38.501400 5003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:38.502131 5003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d6d24987df40b8b8944bc3da2b8028: Sending DeleteTablet for 3 replicas of tablet 319f136312b249449ee48241866cda58
I20260929 18:58:38.502478 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f136312b249449ee48241866cda58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:38 UTC) from {username='slave'} at 127.0.0.1:47602
I20260929 18:58:38.502483 5387 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f136312b249449ee48241866cda58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:38 UTC) from {username='slave'} at 127.0.0.1:34016
I20260929 18:58:38.502511 5125 tablet_service.cc:1514] Processing DeleteTablet for tablet 319f136312b249449ee48241866cda58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:38 UTC) from {username='slave'} at 127.0.0.1:49236
I20260929 18:58:38.502686 5491 tablet_replica.cc:330] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: stopping tablet replica
I20260929 18:58:38.502784 5491 raft_consensus.cc:2229] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:38.502885 5491 raft_consensus.cc:2258] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:38.503031 5492 tablet_replica.cc:330] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: stopping tablet replica
I20260929 18:58:38.503118 5492 raft_consensus.cc:2229] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:38.503233 5492 raft_consensus.cc:2258] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:38.503561 5491 ts_tablet_manager.cc:1905] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:38.504843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5063
I20260929 18:58:38.505062 5492 ts_tablet_manager.cc:1905] T 319f136312b249449ee48241866cda58 P 55dc6aa61c694d40a7757b80f4ee727f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:38.507455 5491 ts_tablet_manager.cc:1918] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:38.507514 5491 log.cc:1196] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/319f136312b249449ee48241866cda58
I20260929 18:58:38.507772 5491 ts_tablet_manager.cc:1939] T 319f136312b249449ee48241866cda58 P b51615abda2d4999ac6805bac2da240a: Deleting consensus metadata
I20260929 18:58:38.508193 4988 catalog_manager.cc:4841] TS b51615abda2d4999ac6805bac2da240a (127.0.98.3:34173): tablet 319f136312b249449ee48241866cda58 (table test-table [id=d23798eb03ac4fe1950427d4aa09079a]) successfully deleted
W20260929 18:58:38.511040 4987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44953 (error 108)
I20260929 18:58:38.511116 5490 tablet_replica.cc:330] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: stopping tablet replica
I20260929 18:58:38.511240 5490 raft_consensus.cc:2229] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:38.511418 5490 raft_consensus.cc:2258] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:38.511849 5490 ts_tablet_manager.cc:1905] T 319f136312b249449ee48241866cda58 P ca1583a9f7514971be9ad73bc060b736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:38.511863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5194
I20260929 18:58:38.511896 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.511744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44953 (local address 127.0.0.1:49242)
0929 18:58:38.511810 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.511860 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:58:38.513371 4987 catalog_manager.cc:4584] TS 55dc6aa61c694d40a7757b80f4ee727f (127.0.98.1:44953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 319f136312b249449ee48241866cda58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44953: connect: Connection refused (error 111)
I20260929 18:58:38.516861 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.516738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36471 (local address 127.0.0.1:47620)
0929 18:58:38.516794 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.516823 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:58:38.516878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5325
I20260929 18:58:38.521430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 4972
2026-09-29T18:58:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (817 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:58:38.541193 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:58:38.542208 392 webserver.cc:460] Webserver started at https://127.0.0.1:45951/ using document root <none> and password file <none>
I20260929 18:58:38.542294 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:58:38.542420 392 webserver.cc:460] Webserver started at http://127.0.0.1:41719/ using document root <none> and password file <none>
2026-09-29T18:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:38Z Disabled control of system clock
I20260929 18:58:38.549934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/testchainca.cert with env {}
W20260929 18:58:38.630214 5499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:38.630422 5499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:38.630443 5499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:38.631701 5499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:38.631740 5499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:38.631752 5499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:38.631765 5499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:38.631778 5499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:58:38.631821 5499 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:58:38.631852 5499 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45951/jwks.json
I20260929 18:58:38.633415 5499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:38.633669 5499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:38.633852 5499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:38.638949 5505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.638973 5504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:38.638995 5499 server_base.cc:1034] running on GCE node
W20260929 18:58:38.638949 5507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:38.639514 5499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:38.639777 5499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:38.640924 5499 hybrid_clock.cc:648] HybridClock initialized: now 1790708318640902 us; error 41 us; skew 500 ppm
I20260929 18:58:38.642310 5499 webserver.cc:460] Webserver started at http://127.0.98.62:37747/ using document root <none> and password file <none>
I20260929 18:58:38.642601 5499 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:38.642652 5499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:38.642809 5499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:38.643919 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "f30820fbfc194f3f934c800afc5b0cc7"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "0b5993cd6844cbe6c804036386680bfb"
server_key_iv: "06d8daaad6c78728b673f915ea56a848"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.644282 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "f30820fbfc194f3f934c800afc5b0cc7"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "0b5993cd6844cbe6c804036386680bfb"
server_key_iv: "06d8daaad6c78728b673f915ea56a848"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.645769 5499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:38.646559 5513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.646772 5499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:38.646859 5499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "f30820fbfc194f3f934c800afc5b0cc7"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "0b5993cd6844cbe6c804036386680bfb"
server_key_iv: "06d8daaad6c78728b673f915ea56a848"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.646914 5499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:38.663432 5499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:38.663733 5499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:38.663837 5499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:38.667593 5499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41431
I20260929 18:58:38.667649 5565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41431 every 8 connection(s)
I20260929 18:58:38.667987 5499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:38.669052 5566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:38.671532 5566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7: Bootstrap starting.
I20260929 18:58:38.672228 5566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:38.672610 5566 log.cc:824] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:38.673403 5566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7: No bootstrap required, opened a new log
I20260929 18:58:38.674360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5499
I20260929 18:58:38.674440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:38.674643 392 external_mini_cluster.cc:1428] Setting key 2173b9e7426ee1cce22e2949ac4221d1
I20260929 18:58:38.675212 5566 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } }
I20260929 18:58:38.675446 5566 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:38.675500 5566 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30820fbfc194f3f934c800afc5b0cc7, State: Initialized, Role: FOLLOWER
I20260929 18:58:38.675650 5566 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } }
I20260929 18:58:38.675752 5566 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:38.675801 5566 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:38.675853 5566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:38.676433 5566 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } }
I20260929 18:58:38.676520 5566 leader_election.cc:302] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30820fbfc194f3f934c800afc5b0cc7; no voters:
I20260929 18:58:38.676707 5566 leader_election.cc:288] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:38.676738 5571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:38.676846 5571 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [term 1 LEADER]: Becoming Leader. State: Replica: f30820fbfc194f3f934c800afc5b0cc7, State: Running, Role: LEADER
I20260929 18:58:38.676957 5566 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:38.676928 5571 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } }
I20260929 18:58:38.677237 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30820fbfc194f3f934c800afc5b0cc7. Latest consensus state: current_term: 1 leader_uuid: "f30820fbfc194f3f934c800afc5b0cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } } }
I20260929 18:58:38.677250 5573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30820fbfc194f3f934c800afc5b0cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30820fbfc194f3f934c800afc5b0cc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41431 } } }
I20260929 18:58:38.677317 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:38.677318 5573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:38.677572 5575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:38.678128 5575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:38.680668 5575 catalog_manager.cc:1347] Generated new cluster ID: 9fcc599e1d6344fbbcb5554aa0f1f8b9
I20260929 18:58:38.680876 5575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:38.681104 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.675199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.0.98.62:41431)
0929 18:58:38.675402 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.675410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.675434 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:58:38.675570 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.675578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.675647 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.675761 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.676079 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.677754 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.678958 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.679424 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.679573 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.680266 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.680270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.680270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.680478 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.680520 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.680535 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.680787 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.680819 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.680893 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":112,"threads_started":1}
I20260929 18:58:38.681963 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:58:38.707471 5575 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:38.708043 5575 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:38.713438 5575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30820fbfc194f3f934c800afc5b0cc7: Generated new TSK 0
I20260929 18:58:38.713662 5575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:58:38.739461 5596 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:58:38.739579 5596 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 18:58:38.740068 5595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:38.723656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41431 (local address 127.0.0.1:48908)
0929 18:58:38.723861 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.723873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.723975 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.724298 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.724300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.724302 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:58:38.724390 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:38.724392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.725137 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.725139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:38.725785 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:38.725787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.726199 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.726203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:38.726273 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.726302 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:58:38.739752 (+ 13450us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:58:38.739841 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:58:38.740388 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.723755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.0.98.62:41431)
0929 18:58:38.723807 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.723810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.723951 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:58:38.724175 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.724176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.724189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.724286 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:58:38.724471 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.725023 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.725880 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.726105 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.726199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.726460 (+ 261us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:58:38.739597 (+ 13137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:58:38.740320 (+ 723us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:58:38.740406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5499
2026-09-29T18:58:38Z chronyd exiting
I20260929 18:58:38.776638 392 test_util.cc:175] -----------------------------------------------
I20260929 18:58:38.776731 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708293967505-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:58:38.777444 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:58:38.778095 392 webserver.cc:460] Webserver started at https://127.0.0.1:43167/ using document root <none> and password file <none>
I20260929 18:58:38.778136 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:58:38.778280 392 webserver.cc:460] Webserver started at http://127.0.0.1:33599/ using document root <none> and password file <none>
2026-09-29T18:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:38Z Disabled control of system clock
I20260929 18:58:38.785601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36675
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/testchainca.cert with env {}
W20260929 18:58:38.865077 5602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:38.865334 5602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:38.865372 5602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:38.866534 5602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:38.866600 5602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:38.866623 5602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:38.866652 5602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:38.866683 5602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:58:38.866705 5602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:58:38.866724 5602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43167/jwks.json
I20260929 18:58:38.868250 5602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:38.868528 5602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:38.868929 5602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:38.874194 5608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.874190 5610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:38.874193 5607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:38.874557 5602 server_base.cc:1034] running on GCE node
I20260929 18:58:38.874770 5602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:38.874998 5602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:38.876138 5602 hybrid_clock.cc:648] HybridClock initialized: now 1790708318876126 us; error 30 us; skew 500 ppm
I20260929 18:58:38.877493 5602 webserver.cc:460] Webserver started at http://127.0.98.62:33353/ using document root <none> and password file <none>
I20260929 18:58:38.877784 5602 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:38.877835 5602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:38.878000 5602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:38.879037 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "573e206dff434a70956ab717d70f7e6c"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "21ec3f570c6901083cb2d4eec73cfc8d"
server_key_iv: "85495eff6fdbc471e0c6811d25be8aea"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.879431 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "573e206dff434a70956ab717d70f7e6c"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "21ec3f570c6901083cb2d4eec73cfc8d"
server_key_iv: "85495eff6fdbc471e0c6811d25be8aea"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.880719 5602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:38.881448 5616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:38.881628 5602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:38.881722 5602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "573e206dff434a70956ab717d70f7e6c"
format_stamp: "Formatted at 2026-09-29 18:58:38 on dist-test-slave-gfsd"
server_key: "21ec3f570c6901083cb2d4eec73cfc8d"
server_key_iv: "85495eff6fdbc471e0c6811d25be8aea"
server_key_version: "encryptionkey@0"
I20260929 18:58:38.881800 5602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:38.908771 5602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:38.909130 5602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:38.909292 5602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:38.913203 5602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36675
I20260929 18:58:38.913241 5668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36675 every 8 connection(s)
I20260929 18:58:38.913611 5602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:38.914630 5669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:38.917162 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c: Bootstrap starting.
I20260929 18:58:38.917724 5669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:38.917982 5669 log.cc:824] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:38.918695 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c: No bootstrap required, opened a new log
I20260929 18:58:38.919993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5602
I20260929 18:58:38.920111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:38.920310 392 external_mini_cluster.cc:1428] Setting key 0bc6157d26432b221698fec4ed16d6a7
I20260929 18:58:38.920611 5669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } }
I20260929 18:58:38.920812 5669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:38.920864 5669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573e206dff434a70956ab717d70f7e6c, State: Initialized, Role: FOLLOWER
I20260929 18:58:38.921005 5669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } }
I20260929 18:58:38.921113 5669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:38.921149 5669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:38.921204 5669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:38.921917 5669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } }
I20260929 18:58:38.922041 5669 leader_election.cc:302] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 573e206dff434a70956ab717d70f7e6c; no voters:
I20260929 18:58:38.922196 5669 leader_election.cc:288] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:38.922439 5669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:38.922461 5674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:38.922654 5674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [term 1 LEADER]: Becoming Leader. State: Replica: 573e206dff434a70956ab717d70f7e6c, State: Running, Role: LEADER
I20260929 18:58:38.922737 5674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } }
I20260929 18:58:38.923158 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "573e206dff434a70956ab717d70f7e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } } }
I20260929 18:58:38.923249 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:38.923444 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573e206dff434a70956ab717d70f7e6c. Latest consensus state: current_term: 1 leader_uuid: "573e206dff434a70956ab717d70f7e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e206dff434a70956ab717d70f7e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36675 } } }
I20260929 18:58:38.923512 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:38.923856 5683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:38.924368 5683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:38.926097 5683 catalog_manager.cc:1347] Generated new cluster ID: fe055cdd518647d79c9f1f811877f5ce
I20260929 18:58:38.926149 5683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:38.926838 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.920878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.98.62:36675)
0929 18:58:38.921138 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.921148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.921292 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:58:38.921445 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.921454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.921554 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.921709 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:38.921879 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.923332 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.924475 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.925107 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.925238 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.926065 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:38.926069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:38.926069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:38.926294 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:38.926357 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:38.926372 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:38.926447 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:38.926551 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 18:58:38.926631 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":116,"threads_started":1}
I20260929 18:58:38.927783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:58:38.932847 5683 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:38.933331 5683 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:38.942756 5683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 573e206dff434a70956ab717d70f7e6c: Generated new TSK 0
I20260929 18:58:38.942966 5683 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:58:38.972667 5699 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:58:38.972760 5699 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:58:38.973052 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:38.950555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.98.62:36675)
0929 18:58:38.950609 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:38.950611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:38.950702 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:58:38.950773 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:38.950774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:38.950784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:38.950844 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:58:38.950978 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.951576 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.952740 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.952959 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.953027 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.953244 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:58:38.972839 (+ 19595us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:58:38.972992 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 18:58:38.973125 5673 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:58:38.973186 5698 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:38.950484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36675 (local address 127.0.0.1:36044)
0929 18:58:38.950637 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:38.950647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:38.950707 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:38.950846 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:38.950847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:38.950848 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:58:38.950919 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:38.950921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.951685 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.951686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:38.952631 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:38.952634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:38.953042 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:38.953047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:38.953122 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:38.953128 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:58:38.972969 (+ 19841us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:58:38.973089 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260929 18:58:38.973671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5602
2026-09-29T18:58:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:58:39.011674 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:58:39.012372 392 webserver.cc:460] Webserver started at https://127.0.0.1:35219/ using document root <none> and password file <none>
I20260929 18:58:39.012440 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:58:39.012600 392 webserver.cc:460] Webserver started at http://127.0.0.1:40561/ using document root <none> and password file <none>
2026-09-29T18:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:39Z Disabled control of system clock
I20260929 18:58:39.020639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43133
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35219/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:58:39.099900 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:39.100154 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:39.100180 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:39.101491 5705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:39.101538 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:39.101552 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:39.101564 5705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:39.101576 5705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:58:39.101625 5705 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:58:39.101656 5705 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35219/jwks.json
I20260929 18:58:39.103077 5705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35219/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:39.103374 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:39.103685 5705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:39.108650 5711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:39.108697 5710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:39.108649 5713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:39.108992 5705 server_base.cc:1034] running on GCE node
I20260929 18:58:39.109324 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:39.109619 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:39.110780 5705 hybrid_clock.cc:648] HybridClock initialized: now 1790708319110767 us; error 45 us; skew 500 ppm
I20260929 18:58:39.112136 5705 webserver.cc:460] Webserver started at http://127.0.98.62:43351/ using document root <none> and password file <none>
I20260929 18:58:39.112393 5705 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:39.112464 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:39.112620 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:39.113690 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "4de19c019db446669b762cbcc81b9478"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "230deaa3d5f8acfb386144148650dc71"
server_key_iv: "949581be3119a5226d38fcfb8306dca6"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.114076 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "4de19c019db446669b762cbcc81b9478"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "230deaa3d5f8acfb386144148650dc71"
server_key_iv: "949581be3119a5226d38fcfb8306dca6"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.115461 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:39.116324 5719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:39.116514 5705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:39.116614 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "4de19c019db446669b762cbcc81b9478"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "230deaa3d5f8acfb386144148650dc71"
server_key_iv: "949581be3119a5226d38fcfb8306dca6"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.116705 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:39.142805 5705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:39.143262 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:39.143461 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:39.147457 5705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43133
I20260929 18:58:39.147500 5771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43133 every 8 connection(s)
I20260929 18:58:39.147856 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:39.148576 5772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:39.151377 5772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478: Bootstrap starting.
I20260929 18:58:39.151973 5772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:39.152268 5772 log.cc:824] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:39.152932 5772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478: No bootstrap required, opened a new log
I20260929 18:58:39.154567 5772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260929 18:58:39.154704 5772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:39.154759 5772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de19c019db446669b762cbcc81b9478, State: Initialized, Role: FOLLOWER
I20260929 18:58:39.154953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5705
I20260929 18:58:39.154927 5772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260929 18:58:39.155014 5772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:39.155053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:39.155063 5772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:39.155109 5772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:39.155264 392 external_mini_cluster.cc:1428] Setting key 0927c089ffd286d1124b6e3eac7af65b
I20260929 18:58:39.155757 5772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260929 18:58:39.155884 5772 leader_election.cc:302] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de19c019db446669b762cbcc81b9478; no voters:
I20260929 18:58:39.156035 5772 leader_election.cc:288] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:39.156070 5777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:39.156207 5777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [term 1 LEADER]: Becoming Leader. State: Replica: 4de19c019db446669b762cbcc81b9478, State: Running, Role: LEADER
I20260929 18:58:39.156296 5772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:39.156293 5777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260929 18:58:39.157692 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de19c019db446669b762cbcc81b9478. Latest consensus state: current_term: 1 leader_uuid: "4de19c019db446669b762cbcc81b9478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } } }
I20260929 18:58:39.157780 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:39.157874 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de19c019db446669b762cbcc81b9478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de19c019db446669b762cbcc81b9478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } } }
I20260929 18:58:39.157965 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:39.157994 5792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:39.158424 5792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:39.160029 5792 catalog_manager.cc:1347] Generated new cluster ID: 5d0d5989c3e54ed09e9fb0fcad2acf24
I20260929 18:58:39.160084 5792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:39.162010 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:39.155801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.98.62:43133)
0929 18:58:39.156150 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:39.156160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:39.156198 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:58:39.156317 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:39.156325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:39.156391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:39.156530 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:39.156784 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.158531 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.159778 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.160417 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.160549 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.161290 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:39.161294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:39.161294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:39.161502 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:39.161563 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:39.161578 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:39.161653 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:39.161721 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:58:39.161800 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260929 18:58:39.164224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:58:39.192692 5792 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:39.193259 5792 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:39.196630 5792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de19c019db446669b762cbcc81b9478: Generated new TSK 0
I20260929 18:58:39.196825 5792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:58:39.214140 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:39.210566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.0.98.62:43133)
0929 18:58:39.210613 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:39.210616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:39.210720 (+ 104us) server_negotiation.cc:407] Connection header received
0929 18:58:39.210780 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:39.210781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:39.210792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:39.210850 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:58:39.211001 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.211545 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.212663 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.212880 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.212938 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.213121 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:58:39.213878 (+ 757us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35219/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:58:39.214029 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35219/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 18:58:39.214161 5801 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:39.210476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43133 (local address 127.0.0.1:54084)
0929 18:58:39.210640 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:39.210652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:39.210720 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:39.210866 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:39.210867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:39.210869 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:58:39.210932 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:39.210935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.211625 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.211626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:39.212561 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:39.212564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.212959 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.212966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:39.213028 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.213034 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:58:39.213986 (+ 952us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35219/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:58:39.214071 (+ 85us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35219/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
W20260929 18:58:39.214215 5776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35219/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260929 18:58:39.214321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5705
E20260929 18:58:39.225441 5802 webserver.cc:533] Webserver: sslize failed:
E20260929 18:58:39.225517 5802 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:58:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:39Z Disabled control of system clock
I20260929 18:58:39.259218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36317
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/testchainca.cert with env {}
W20260929 18:58:39.338907 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:39.339114 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:39.339135 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:39.340231 5806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:39.340267 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:39.340281 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:39.340292 5806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:39.340304 5806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:58:39.340317 5806 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:58:39.340327 5806 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:58:39.341709 5806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:39.341892 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:39.342051 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:39.346871 5812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:39.346833 5811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:39.346983 5814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:39.347123 5806 server_base.cc:1034] running on GCE node
I20260929 18:58:39.347393 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:39.347659 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:39.348821 5806 hybrid_clock.cc:648] HybridClock initialized: now 1790708319348809 us; error 27 us; skew 500 ppm
I20260929 18:58:39.350203 5806 webserver.cc:460] Webserver started at http://127.0.98.62:37919/ using document root <none> and password file <none>
I20260929 18:58:39.350452 5806 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:39.350498 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:39.350656 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:39.351758 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "3d068ca2796a4acc882755fa982710f6"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "bf4939a04769ad83f02bf89b05003bd1"
server_key_iv: "6a4832b729ac58dbc8557c0c45c27ce9"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.352109 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d068ca2796a4acc882755fa982710f6"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "bf4939a04769ad83f02bf89b05003bd1"
server_key_iv: "6a4832b729ac58dbc8557c0c45c27ce9"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.353394 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:39.354174 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:39.354358 5806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:39.354431 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "3d068ca2796a4acc882755fa982710f6"
format_stamp: "Formatted at 2026-09-29 18:58:39 on dist-test-slave-gfsd"
server_key: "bf4939a04769ad83f02bf89b05003bd1"
server_key_iv: "6a4832b729ac58dbc8557c0c45c27ce9"
server_key_version: "encryptionkey@0"
I20260929 18:58:39.354519 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:39.375960 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:39.376303 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:39.376461 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:39.380822 5806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36317
I20260929 18:58:39.380889 5872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36317 every 8 connection(s)
I20260929 18:58:39.381253 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:39.382191 5873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:39.383746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5806
I20260929 18:58:39.383836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:39.384027 392 external_mini_cluster.cc:1428] Setting key 9563138a6d4387a9da01d2b12f2a11fb
I20260929 18:58:39.384769 5873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6: Bootstrap starting.
I20260929 18:58:39.385365 5873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:39.385736 5873 log.cc:824] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:39.386487 5873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6: No bootstrap required, opened a new log
I20260929 18:58:39.389046 5873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260929 18:58:39.389235 5873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:39.389261 5873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d068ca2796a4acc882755fa982710f6, State: Initialized, Role: FOLLOWER
I20260929 18:58:39.389393 5873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260929 18:58:39.389465 5873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:39.389505 5873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:39.389564 5873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:39.390069 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:39.384561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.98.62:36317)
0929 18:58:39.384748 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:39.384755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:39.384780 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:58:39.385228 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:39.385235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:39.385293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:39.385408 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:39.385657 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.386754 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.387892 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.388291 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.388407 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.389178 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:39.389182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:39.389182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:58:39.389442 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:39.389509 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:58:39.389526 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:39.389608 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:39.389819 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:58:39.389901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 18:58:39.390093 5873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260929 18:58:39.390229 5873 leader_election.cc:302] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d068ca2796a4acc882755fa982710f6; no voters:
I20260929 18:58:39.390384 5873 leader_election.cc:288] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:39.390422 5878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:39.390596 5878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [term 1 LEADER]: Becoming Leader. State: Replica: 3d068ca2796a4acc882755fa982710f6, State: Running, Role: LEADER
I20260929 18:58:39.390619 5873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:39.390689 5878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260929 18:58:39.391016 5878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d068ca2796a4acc882755fa982710f6. Latest consensus state: current_term: 1 leader_uuid: "3d068ca2796a4acc882755fa982710f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260929 18:58:39.391121 5878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:39.391026 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d068ca2796a4acc882755fa982710f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d068ca2796a4acc882755fa982710f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260929 18:58:39.391304 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:39.391347 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:58:39.391433 5887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:39.391945 5887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:39.393472 5887 catalog_manager.cc:1347] Generated new cluster ID: 70bd7a41b561422da973ecbc70e5ac53
I20260929 18:58:39.393528 5887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:39.402321 5887 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:39.402848 5887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:39.416400 5887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d068ca2796a4acc882755fa982710f6: Generated new TSK 0
I20260929 18:58:39.416649 5887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:58:39.873296 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:39.421485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.98.62:36317)
0929 18:58:39.421533 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:39.421535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:39.421625 (+ 90us) server_negotiation.cc:407] Connection header received
0929 18:58:39.421685 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:39.421686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:39.421700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:39.421780 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:58:39.421994 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.422785 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.423772 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.424076 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.424157 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.424309 (+ 152us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:58:39.873057 (+448748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:58:39.873231 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51406: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 18:58:39.873363 5876 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51406: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:58:39.873623 5902 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:39.421403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.0.1:51406)
0929 18:58:39.421539 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:39.421550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:39.421618 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:39.421859 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:39.421860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:39.421862 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:58:39.421916 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:39.421918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.422885 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.422885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:39.423693 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:39.423695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:39.424160 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:39.424165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:39.424211 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:39.424215 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:58:39.873220 (+449005us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:58:39.873482 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260929 18:58:39.873957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5806
2026-09-29T18:58:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (640 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:40 dist-test-slave-gfsd krb5kdc[5914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:40 dist-test-slave-gfsd krb5kdc[5914](info): set up 2 sockets
Sep 29 18:58:40 dist-test-slave-gfsd krb5kdc[5914](info): commencing operation
krb5kdc: starting...
W20260929 18:58:41.087502 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:41.205598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:41.286829 5927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:41.287031 5927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:41.287052 5927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:41.288141 5927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:41.288177 5927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:41.288213 5927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:41.288226 5927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:41.288239 5927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:41.289747 5927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:41.289939 5927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:41.290099 5927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:41.294955 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.294961 5927 server_base.cc:1034] running on GCE node
W20260929 18:58:41.294905 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.294919 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.295439 5927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:41.295691 5927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:41.296855 5927 hybrid_clock.cc:648] HybridClock initialized: now 1790708321296848 us; error 43 us; skew 500 ppm
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.299400 5927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:41.299886 5927 webserver.cc:460] Webserver started at http://127.0.98.62:46291/ using document root <none> and password file <none>
I20260929 18:58:41.300124 5927 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:41.300192 5927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:41.300336 5927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:41.301424 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "768f2d487f9b4b38a2f5d47ec077cc72"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "ff03e98263512f510aeea5db4b03649e"
server_key_iv: "bb00b45b31b3c40f00eb1a6090f6d822"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.301800 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "768f2d487f9b4b38a2f5d47ec077cc72"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "ff03e98263512f510aeea5db4b03649e"
server_key_iv: "bb00b45b31b3c40f00eb1a6090f6d822"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.303151 5927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:58:41.304069 5942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.304345 5927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:41.304457 5927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "768f2d487f9b4b38a2f5d47ec077cc72"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "ff03e98263512f510aeea5db4b03649e"
server_key_iv: "bb00b45b31b3c40f00eb1a6090f6d822"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.304561 5927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:41.318363 5927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:41.320498 5927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:41.320675 5927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:41.324970 5927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45933
I20260929 18:58:41.325013 5994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45933 every 8 connection(s)
I20260929 18:58:41.325512 5927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:41.326198 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:41.328433 5995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72: Bootstrap starting.
I20260929 18:58:41.328938 5995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:41.329169 5995 log.cc:824] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:41.329761 5995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72: No bootstrap required, opened a new log
I20260929 18:58:41.330286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 5927
I20260929 18:58:41.330400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:41.330597 392 external_mini_cluster.cc:1428] Setting key d529c3a8497b057b20c48ff161294eb4
I20260929 18:58:41.331918 5995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } }
I20260929 18:58:41.332084 5995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:41.332123 5995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768f2d487f9b4b38a2f5d47ec077cc72, State: Initialized, Role: FOLLOWER
I20260929 18:58:41.332311 5995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } }
I20260929 18:58:41.332432 5995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:41.332468 5995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:41.332547 5995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:41.333181 5995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } }
I20260929 18:58:41.333264 5995 leader_election.cc:302] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 768f2d487f9b4b38a2f5d47ec077cc72; no voters:
I20260929 18:58:41.333432 5995 leader_election.cc:288] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:41.333478 6000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:41.333690 6000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [term 1 LEADER]: Becoming Leader. State: Replica: 768f2d487f9b4b38a2f5d47ec077cc72, State: Running, Role: LEADER
I20260929 18:58:41.333702 5995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:41.333860 6000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } }
I20260929 18:58:41.334827 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } } }
I20260929 18:58:41.334911 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:41.335177 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:41.335403 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 768f2d487f9b4b38a2f5d47ec077cc72. Latest consensus state: current_term: 1 leader_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768f2d487f9b4b38a2f5d47ec077cc72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45933 } } }
I20260929 18:58:41.335656 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:41.336125 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:41.337742 6013 catalog_manager.cc:1347] Generated new cluster ID: 9a62d7facf01488b9df0869fdb5fc2f1
I20260929 18:58:41.337798 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:41.341949 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.331122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.62:45933)
0929 18:58:41.331272 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.331321 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.331356 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:58:41.331900 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.331906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.331950 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.332139 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.332904 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.334251 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.335408 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.335951 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.336092 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.338724 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.338729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.338729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.338740 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.340485 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.340855 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.340858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.340859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.340961 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.341172 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.341173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.341174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.341405 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.341568 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.341646 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.341782 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260929 18:58:41.344432 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:41.345003 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:41.355439 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 768f2d487f9b4b38a2f5d47ec077cc72: Generated new TSK 0
I20260929 18:58:41.355638 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:41.389890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:41.476998 6021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:41.477259 6021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:41.477299 6021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:41.478529 6021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:41.478605 6021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:41.478653 6021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:41.480221 6021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:41.480494 6021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:41.480744 6021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:41.486048 6029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.486048 6026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.486186 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.486425 6021 server_base.cc:1034] running on GCE node
I20260929 18:58:41.486599 6021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:41.486819 6021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:41.487953 6021 hybrid_clock.cc:648] HybridClock initialized: now 1790708321487934 us; error 33 us; skew 500 ppm
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.490547 6021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:41.490974 6021 webserver.cc:460] Webserver started at http://127.0.98.1:42501/ using document root <none> and password file <none>
I20260929 18:58:41.491228 6021 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:41.491313 6021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:41.491487 6021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:41.492542 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "78477c78da3041fdb9b104188089cdb7"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "df981e46f1a8417b5fd47ba72319a9e6"
server_key_iv: "3cacd136cca37ea60caf627e76d6bb3e"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.492911 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "78477c78da3041fdb9b104188089cdb7"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "df981e46f1a8417b5fd47ba72319a9e6"
server_key_iv: "3cacd136cca37ea60caf627e76d6bb3e"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.494220 6021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:41.495143 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.495538 6021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:41.495642 6021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "78477c78da3041fdb9b104188089cdb7"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "df981e46f1a8417b5fd47ba72319a9e6"
server_key_iv: "3cacd136cca37ea60caf627e76d6bb3e"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.495718 6021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:41.544090 6021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:41.546280 6021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:41.546435 6021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:41.546701 6021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:41.547032 6021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:41.547086 6021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.547128 6021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:41.547165 6021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.553046 6021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42423
I20260929 18:58:41.553237 6149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42423 every 8 connection(s)
I20260929 18:58:41.553539 6021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:41.555821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6021
I20260929 18:58:41.555959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:41.556143 392 external_mini_cluster.cc:1428] Setting key f5b2346cdb826b5175fe518d093383cc
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:41.563921 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.554335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60105 (local address 127.0.98.62:45933)
0929 18:58:41.554379 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.554381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.554599 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:58:41.555551 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.555552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.555569 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.555673 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.556394 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.557013 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.558072 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.558305 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.558366 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.560858 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.560859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.560860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.560868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.562388 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.562887 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.562888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.562888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.562941 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.563224 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.563224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.563225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.563386 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.563446 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.563784 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.563853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:41.564193 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.554320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45933 (local address 127.0.98.1:60105)
0929 18:58:41.554469 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:41.554510 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:41.555458 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:41.555726 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:41.555729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:41.556032 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:41.556289 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.556295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.557107 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.557108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.557967 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.557970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.558388 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.558389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.558536 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.559039 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:41.559049 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:41.560761 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:41.562482 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.562490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.562493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.562808 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.563008 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.563009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.563009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.563151 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.563456 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:41.563463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:41.563631 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 18:58:41.563764 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 18:58:41.563937 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260929 18:58:41.564802 6150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45933
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:58:41.564915 6150 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:41.565112 6150 heartbeater.cc:511] Master 127.0.98.62:45933 requested a full tablet report, sending...
I20260929 18:58:41.565658 5959 ts_manager.cc:194] Registered new tserver with Master: 78477c78da3041fdb9b104188089cdb7 (127.0.98.1:42423)
I20260929 18:58:41.566421 5959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:41.582597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:41.662901 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:41.663151 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:41.663193 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:41.664376 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:41.664448 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:41.664490 6155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:41.665921 6155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:41.666191 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:41.666421 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:41.671602 6161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.671552 6160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.671543 6163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.671890 6155 server_base.cc:1034] running on GCE node
I20260929 18:58:41.672098 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:41.672334 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:41.673512 6155 hybrid_clock.cc:648] HybridClock initialized: now 1790708321673486 us; error 40 us; skew 500 ppm
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.676010 6155 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:41.676376 6155 webserver.cc:460] Webserver started at http://127.0.98.2:40513/ using document root <none> and password file <none>
I20260929 18:58:41.676573 6155 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:41.676615 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:41.676787 6155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:41.677774 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5fd189d86b1407fb8236d735dbc188f"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "8c93135ad22d986b103c5041ed21cf95"
server_key_iv: "618e5b456dbc3044b9811dd207fd096f"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.678076 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5fd189d86b1407fb8236d735dbc188f"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "8c93135ad22d986b103c5041ed21cf95"
server_key_iv: "618e5b456dbc3044b9811dd207fd096f"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.679371 6155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:41.680063 6170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.680230 6155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:41.680325 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "e5fd189d86b1407fb8236d735dbc188f"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "8c93135ad22d986b103c5041ed21cf95"
server_key_iv: "618e5b456dbc3044b9811dd207fd096f"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.680408 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:41.697402 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:41.699739 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:41.699898 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:41.700125 6155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:41.700420 6155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:41.700474 6155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.700524 6155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:41.700568 6155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.706116 6155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37189
I20260929 18:58:41.706213 6283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37189 every 8 connection(s)
I20260929 18:58:41.706519 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:41.707163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6155
I20260929 18:58:41.707273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:41.707490 392 external_mini_cluster.cc:1428] Setting key a6b93970f807b2413a167a6bc70be5bf
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:58:41.717583 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.707332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37641 (local address 127.0.98.62:45933)
0929 18:58:41.707378 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.707380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.707606 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:58:41.708839 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.708841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.708852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.708906 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.709641 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.710164 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.711800 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.711996 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.712060 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.714594 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.714596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.714596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.714603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.716156 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.716622 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.716623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.716624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.716685 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.716939 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.716940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.716940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.717061 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.717138 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.717410 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.717493 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:58:41.717780 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.707347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45933 (local address 127.0.98.2:37641)
0929 18:58:41.707488 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:41.707526 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:41.708736 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:41.708977 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:41.708982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:41.709232 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:41.709532 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.709539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.710258 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.710259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.711708 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.711712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.712087 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.712088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.712235 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.712712 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:41.712723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:41.714461 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:41.716252 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.716258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.716260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.716518 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.716753 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.716754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.716754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.716867 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.717130 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:41.717136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:41.717280 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:58:41.717389 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:58:41.717542 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:58:41.718319 6284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45933
I20260929 18:58:41.718417 6284 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:41.718601 6284 heartbeater.cc:511] Master 127.0.98.62:45933 requested a full tablet report, sending...
I20260929 18:58:41.719017 5959 ts_manager.cc:194] Registered new tserver with Master: e5fd189d86b1407fb8236d735dbc188f (127.0.98.2:37189)
I20260929 18:58:41.719560 5959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:41.733508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:41.815541 6289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:41.815724 6289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:41.815747 6289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:41.816885 6289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:41.816926 6289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:41.816951 6289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:41.818403 6289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:41.818624 6289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:41.818846 6289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:41.823794 6295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:41.823850 6294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.823985 6289 server_base.cc:1034] running on GCE node
W20260929 18:58:41.824026 6297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:41.824251 6289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:41.824477 6289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:41.825615 6289 hybrid_clock.cc:648] HybridClock initialized: now 1790708321825565 us; error 74 us; skew 500 ppm
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.832259 6289 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:41.832716 6289 webserver.cc:460] Webserver started at http://127.0.98.3:34473/ using document root <none> and password file <none>
I20260929 18:58:41.832937 6289 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:41.833003 6289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:41.833156 6289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:41.834385 6289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed5cbeb2306b4410a76361e8efe1d3b4"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "aa4462e5e7ec9879b9359d19d8a24106"
server_key_iv: "226a3acdfaad1555de47389a5191a8fd"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.834818 6289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed5cbeb2306b4410a76361e8efe1d3b4"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "aa4462e5e7ec9879b9359d19d8a24106"
server_key_iv: "226a3acdfaad1555de47389a5191a8fd"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.836300 6289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:41.837011 6304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.837193 6289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:41.837289 6289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "ed5cbeb2306b4410a76361e8efe1d3b4"
format_stamp: "Formatted at 2026-09-29 18:58:41 on dist-test-slave-gfsd"
server_key: "aa4462e5e7ec9879b9359d19d8a24106"
server_key_iv: "226a3acdfaad1555de47389a5191a8fd"
server_key_version: "encryptionkey@0"
I20260929 18:58:41.837375 6289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:41.858906 6289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:41.860847 6289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:41.860998 6289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:41.861232 6289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:41.861522 6289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:41.861574 6289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.861616 6289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:41.861644 6289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:41.867344 6289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34661
I20260929 18:58:41.867419 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34661 every 8 connection(s)
I20260929 18:58:41.867741 6289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:41.868333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6289
I20260929 18:58:41.868440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:41.868686 392 external_mini_cluster.cc:1428] Setting key 806e48cfcdc6b253931fb733f2886b2c
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:41.877024 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.868921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34489 (local address 127.0.98.62:45933)
0929 18:58:41.868967 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.868970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.868982 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:58:41.869940 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.869941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.869951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.870010 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.870600 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.871155 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.872156 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.872343 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.872400 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.874359 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.874360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.874361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.874367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.875929 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.876286 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.876287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.876288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.876337 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.876534 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.876535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.876535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.876648 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.876694 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.876907 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.876963 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:41.877220 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45933 (local address 127.0.98.3:34489)
0929 18:58:41.868616 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:41.868658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:41.869836 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:41.870067 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:41.870070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:41.870315 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:41.870498 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.870501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.871238 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.871239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.872076 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:41.872078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.872416 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.872417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:41.872512 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.872926 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:41.872933 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:41.874267 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:41.876000 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.876005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.876007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.876223 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.876393 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:41.876393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:41.876393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:41.876482 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:41.876687 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:41.876692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:41.876802 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:58:41.876883 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:58:41.876997 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260929 18:58:41.877748 6418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45933
I20260929 18:58:41.877830 6418 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:41.878019 6418 heartbeater.cc:511] Master 127.0.98.62:45933 requested a full tablet report, sending...
I20260929 18:58:41.878412 5959 ts_manager.cc:194] Registered new tserver with Master: ed5cbeb2306b4410a76361e8efe1d3b4 (127.0.98.3:34661)
I20260929 18:58:41.878801 5959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34489
I20260929 18:58:41.880316 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/test-admin.keytab.
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.895749 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:58:41.897431 6425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:41.896554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.0.98.1:42423)
0929 18:58:41.896724 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.896728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.896738 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:41.896761 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.896762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.896775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.896844 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.897333 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42802: BlockingRecv error: recv got EOF from 127.0.0.1:42802 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260929 18:58:41.897503 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.898551 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.899574 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:41.907665 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.900197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.0.98.1:42423)
0929 18:58:41.900256 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.900258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.900291 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:58:41.900359 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.900360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.900370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.900418 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.900849 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.901575 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.902304 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.902617 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.902672 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.904747 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.904755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.904755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.904761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.906292 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.906703 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.906705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.906706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.906761 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.906951 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.906952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.906952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.907164 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.907273 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.907525 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.907592 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:58:41.909403 6425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:41.908891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.98.1:42423)
0929 18:58:41.908928 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.908930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.908973 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:58:41.909020 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.909020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.909027 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.909084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.909346 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42828: BlockingRecv error: recv got EOF from 127.0.0.1:42828 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 18:58:41.909500 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc)
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.910459 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.911549 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:41.918495 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.912190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.98.1:42423)
0929 18:58:41.912235 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.912237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.912251 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:58:41.912336 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.912336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.912344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.912391 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.912775 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.913253 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.913955 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.914208 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.914262 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.916255 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.916256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.916256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.916262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.917564 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.917864 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.917865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.917865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.917910 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.918068 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.918068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.918069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.918145 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.918208 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.918308 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.918361 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260929 18:58:41.919941 6425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:41.919485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.0.98.1:42423)
0929 18:58:41.919521 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.919522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.919561 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:58:41.919618 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.919619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.919625 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.919668 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.919913 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42842: BlockingRecv error: recv got EOF from 127.0.0.1:42842 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:58:41.920034 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708293967505-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.920938 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:41.921736 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:58:41.927874 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:41.922202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.0.98.1:42423)
0929 18:58:41.922254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:41.922255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:41.922266 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:41.922322 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:41.922322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:41.922329 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:41.922371 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:41.922665 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.923125 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.923817 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:41.923984 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:41.924036 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:41.925887 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:41.925888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:41.925889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:41.925894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:41.927002 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.927348 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.927353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.927353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.927400 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:41.927559 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:41.927560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:41.927560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:41.927634 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:41.927695 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:41.927778 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:58:41.927826 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:58:41.928478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6021
I20260929 18:58:41.933029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6155
I20260929 18:58:41.937573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6289
I20260929 18:58:41.941632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 5927
2026-09-29T18:58:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1894 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[6438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[6438](info): set up 2 sockets
Sep 29 18:58:41 dist-test-slave-gfsd krb5kdc[6438](info): commencing operation
krb5kdc: starting...
W20260929 18:58:42.993404 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 18:58:43.112347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:43.192723 6451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:43.192924 6451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:43.192945 6451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:43.194118 6451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:43.194162 6451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:43.194176 6451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:43.194188 6451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:43.194200 6451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:43.195708 6451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:43.195961 6451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:43.196218 6451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:43.201249 6456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.201330 6451 server_base.cc:1034] running on GCE node
W20260929 18:58:43.201270 6459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.201263 6457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.201807 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:43.202095 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:43.203266 6451 hybrid_clock.cc:648] HybridClock initialized: now 1790708323203239 us; error 37 us; skew 500 ppm
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:43.206140 6451 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 18:58:43.206624 6451 webserver.cc:460] Webserver started at http://127.0.98.62:42555/ using document root <none> and password file <none>
I20260929 18:58:43.206861 6451 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.206929 6451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.207087 6451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:43.208244 6451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "094bdce469b371f5aa0d43dd26827a8e"
server_key_iv: "fe6657e347f94f813e2c4da663f901e5"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.208611 6451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "094bdce469b371f5aa0d43dd26827a8e"
server_key_iv: "fe6657e347f94f813e2c4da663f901e5"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.209941 6451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:43.210697 6466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.210883 6451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:43.210986 6451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "094bdce469b371f5aa0d43dd26827a8e"
server_key_iv: "fe6657e347f94f813e2c4da663f901e5"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.211061 6451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.246238 6451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:43.249136 6451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:43.249301 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:43.253589 6451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39813
I20260929 18:58:43.253624 6518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39813 every 8 connection(s)
I20260929 18:58:43.253997 6451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:43.254994 6519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:43.257126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6451
I20260929 18:58:43.257256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:43.257391 6519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Bootstrap starting.
I20260929 18:58:43.257474 392 external_mini_cluster.cc:1428] Setting key 2361f6ce43995bdf802769f70ca850a4
I20260929 18:58:43.258070 6519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:43.258354 6519 log.cc:824] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:43.259092 6519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: No bootstrap required, opened a new log
I20260929 18:58:43.261291 6519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260929 18:58:43.261451 6519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:43.261482 6519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6b66361c5b4b6c9f6bfbd1db7d15db, State: Initialized, Role: FOLLOWER
I20260929 18:58:43.261644 6519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260929 18:58:43.261731 6519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:43.261754 6519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:43.261827 6519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:43.262370 6519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260929 18:58:43.262475 6519 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6b66361c5b4b6c9f6bfbd1db7d15db; no voters:
I20260929 18:58:43.262655 6519 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:43.262705 6524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:43.262867 6524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 LEADER]: Becoming Leader. State: Replica: 5a6b66361c5b4b6c9f6bfbd1db7d15db, State: Running, Role: LEADER
I20260929 18:58:43.262933 6519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:43.262965 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260929 18:58:43.263252 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6b66361c5b4b6c9f6bfbd1db7d15db. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260929 18:58:43.263347 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:43.264313 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260929 18:58:43.264497 6529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:43.264567 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:43.265269 6529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:43.266867 6529 catalog_manager.cc:1347] Generated new cluster ID: 0a20dd9ee4fd4d9a91ab02a7950853cb
I20260929 18:58:43.266918 6529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:43.269485 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.257985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.98.62:39813)
0929 18:58:43.258202 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.258209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.258235 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:58:43.258725 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.258731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.258779 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.258896 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.259684 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.260554 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.261668 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.262285 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.262367 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.265402 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.265405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.265406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.265418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.267777 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.268250 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.268253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.268254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.268361 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.268565 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.268566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.268567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.268875 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.269050 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.269127 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.269267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260929 18:58:43.278431 6529 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:43.278973 6529 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:43.283115 6529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Generated new TSK 0
I20260929 18:58:43.283332 6529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 18:58:43.312429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:43.392518 6545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:43.392719 6545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:43.392740 6545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:43.393850 6545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:43.393889 6545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:43.393915 6545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:43.395371 6545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:43.395593 6545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:43.395828 6545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:43.400903 6553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.400903 6551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.400929 6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.401275 6545 server_base.cc:1034] running on GCE node
I20260929 18:58:43.401474 6545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:43.401703 6545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:43.402866 6545 hybrid_clock.cc:648] HybridClock initialized: now 1790708323402849 us; error 32 us; skew 500 ppm
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:43.405484 6545 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:58:43.405886 6545 webserver.cc:460] Webserver started at http://127.0.98.1:33863/ using document root <none> and password file <none>
I20260929 18:58:43.406104 6545 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.406177 6545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.406322 6545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:43.407542 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "4285b7f578a0453d9420b21e9346972a"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "5e15a86752796a91d22d5f3b2cb11925"
server_key_iv: "9a602bfe5b36909507afd619395e2f66"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.407895 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4285b7f578a0453d9420b21e9346972a"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "5e15a86752796a91d22d5f3b2cb11925"
server_key_iv: "9a602bfe5b36909507afd619395e2f66"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.409300 6545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:43.410120 6560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.410311 6545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:43.410413 6545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "4285b7f578a0453d9420b21e9346972a"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "5e15a86752796a91d22d5f3b2cb11925"
server_key_iv: "9a602bfe5b36909507afd619395e2f66"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.410483 6545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.432072 6545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:43.434438 6545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:43.434609 6545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:43.434856 6545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:43.435169 6545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:43.435221 6545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.435261 6545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:43.435307 6545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.441033 6545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33879
I20260929 18:58:43.441082 6673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33879 every 8 connection(s)
I20260929 18:58:43.441385 6545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:43.447328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6545
I20260929 18:58:43.447433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:43.447686 392 external_mini_cluster.cc:1428] Setting key 743f824d785340bbf8077511069b330f
I20260929 18:58:43.451808 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.442066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43229 (local address 127.0.98.62:39813)
0929 18:58:43.442119 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.442121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.442341 (+ 220us) server_negotiation.cc:407] Connection header received
0929 18:58:43.442856 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.442858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.442868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.442926 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.443976 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.444497 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.445546 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.445771 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.445861 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.448198 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.448199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.448199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.448206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.449941 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.450598 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.450600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.450600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.450655 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.450950 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.450951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.450951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.451054 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.451205 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.451560 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.451665 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:58:43.451833 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.442030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:43229)
0929 18:58:43.442224 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.442258 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.442756 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.443007 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.443011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.443678 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.443874 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.443880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.444619 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.444620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.445451 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.445453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.445878 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.445879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.446001 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.446480 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.446488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.448117 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.450139 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.450147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.450150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.450501 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.450720 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.450721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.450721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.450874 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.451219 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.451225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.451425 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.451517 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.451657 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260929 18:58:43.452441 6674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:43.452554 6674 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:43.452768 6674 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:43.453281 6483 ts_manager.cc:194] Registered new tserver with Master: 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879)
I20260929 18:58:43.453902 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43229
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 18:58:43.474782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:43.557503 6679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:43.557722 6679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:43.557747 6679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:43.558821 6679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:43.558861 6679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:43.558888 6679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:43.560406 6679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:43.560611 6679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:43.560792 6679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:43.566165 6687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.566197 6685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.566218 6684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.566500 6679 server_base.cc:1034] running on GCE node
I20260929 18:58:43.566761 6679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:43.567077 6679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:43.568291 6679 hybrid_clock.cc:648] HybridClock initialized: now 1790708323568219 us; error 53 us; skew 500 ppm
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:43.570950 6679 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:58:43.571429 6679 webserver.cc:460] Webserver started at http://127.0.98.2:37995/ using document root <none> and password file <none>
I20260929 18:58:43.571645 6679 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.571696 6679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.571790 6679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:43.572808 6679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "a18822a622364ca796a5484927e6ae22"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "e228b2f0a6ddd66e55bbc879c51b4c9f"
server_key_iv: "89fb7bdb595b46b25c2319acdbbf1f51"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.573148 6679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a18822a622364ca796a5484927e6ae22"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "e228b2f0a6ddd66e55bbc879c51b4c9f"
server_key_iv: "89fb7bdb595b46b25c2319acdbbf1f51"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.574512 6679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:58:43.575459 6694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.575628 6679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:43.575716 6679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "a18822a622364ca796a5484927e6ae22"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "e228b2f0a6ddd66e55bbc879c51b4c9f"
server_key_iv: "89fb7bdb595b46b25c2319acdbbf1f51"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.575771 6679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.590710 6679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:43.592799 6679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:43.592908 6679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:43.593083 6679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:43.593415 6679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:43.593449 6679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.593472 6679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:43.593519 6679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.599189 6679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34925
I20260929 18:58:43.599298 6807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34925 every 8 connection(s)
I20260929 18:58:43.599673 6679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:43.608755 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.600441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57073 (local address 127.0.98.62:39813)
0929 18:58:43.600505 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.600507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.600688 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:58:43.601169 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.601170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.601180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.601236 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.602173 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.602650 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.603738 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.603929 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.603992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.606147 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.606148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.606148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.606156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.607449 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.607806 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.607807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.607807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.607859 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.608110 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.608110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.608111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.608212 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.608272 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.608600 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.608664 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:58:43.608907 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.600409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:57073)
0929 18:58:43.600563 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.600604 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.601064 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.601289 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.601293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.601894 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.602087 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.602092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.602732 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.602733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.603657 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.603661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.604009 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.604010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.604161 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.604697 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.604706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.606017 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.607521 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.607526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.607528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.607740 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.607919 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.607920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.607920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.608014 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.608347 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.608353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.608468 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.608571 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.608702 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260929 18:58:43.609472 6808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:43.609567 6808 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:43.609750 6808 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:43.609822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6679
I20260929 18:58:43.609929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:43.610167 392 external_mini_cluster.cc:1428] Setting key c80298da8cf7fc447f91e253ef3166b5
I20260929 18:58:43.610210 6483 ts_manager.cc:194] Registered new tserver with Master: a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:43.610718 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57073
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:58:43.634505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:43.716313 6813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:43.716562 6813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:43.716605 6813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:43.717729 6813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:43.717792 6813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:43.717835 6813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:43.719369 6813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:43.719628 6813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:43.719854 6813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:43.724910 6818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.724936 6821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.724929 6819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.725008 6813 server_base.cc:1034] running on GCE node
I20260929 18:58:43.725350 6813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:43.725533 6813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:43.726683 6813 hybrid_clock.cc:648] HybridClock initialized: now 1790708323726663 us; error 34 us; skew 500 ppm
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:43.729176 6813 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:58:43.729609 6813 webserver.cc:460] Webserver started at http://127.0.98.3:40467/ using document root <none> and password file <none>
I20260929 18:58:43.729785 6813 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.729823 6813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.729908 6813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:43.730906 6813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9fb93a0c7444beb9f83f0180302db3c"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "c553e8e6bd2f2930d248166a6e85f8c5"
server_key_iv: "5f8550ef16f3b10323fa015ea04ae1c6"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.731225 6813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9fb93a0c7444beb9f83f0180302db3c"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "c553e8e6bd2f2930d248166a6e85f8c5"
server_key_iv: "5f8550ef16f3b10323fa015ea04ae1c6"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.732510 6813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:43.733219 6828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.733399 6813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:43.733487 6813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "f9fb93a0c7444beb9f83f0180302db3c"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "c553e8e6bd2f2930d248166a6e85f8c5"
server_key_iv: "5f8550ef16f3b10323fa015ea04ae1c6"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.733552 6813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.742941 6813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:43.744886 6813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:43.744994 6813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:43.745168 6813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:43.745466 6813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:43.745522 6813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.745570 6813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:43.745594 6813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.751125 6813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36747
I20260929 18:58:43.751205 6941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36747 every 8 connection(s)
I20260929 18:58:43.751525 6813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:43.759246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6813
I20260929 18:58:43.759405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:43.759619 392 external_mini_cluster.cc:1428] Setting key ef79c2cc9705031af8623c4044afd2ef
I20260929 18:58:43.761097 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.752254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49311 (local address 127.0.98.62:39813)
0929 18:58:43.752320 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.752322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.752530 (+ 208us) server_negotiation.cc:407] Connection header received
0929 18:58:43.753062 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.753063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.753076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.753129 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.754063 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.754590 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.755748 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.755943 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.756030 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.758108 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.758109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.758110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.758117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.759614 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.760114 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.760115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.760116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.760178 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.760438 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.760439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.760439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.760557 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.760642 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.760929 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.761008 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:58:43.761304 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.752241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:49311)
0929 18:58:43.752407 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.752443 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.752951 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.753205 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.753209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.753773 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.753959 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.753964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.754700 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.754701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.755662 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.755664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.756036 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.756037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.756165 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.756690 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.756698 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.758013 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.759716 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.759722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.759724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.760040 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.760247 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.760247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.760248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.760362 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.760634 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.760640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.760782 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.760905 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.761062 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260929 18:58:43.761780 6942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:43.761860 6942 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:43.761991 6942 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:43.762364 6482 ts_manager.cc:194] Registered new tserver with Master: f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3:36747)
I20260929 18:58:43.762840 6482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49311
I20260929 18:58:43.771569 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 18:58:43.798267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39317
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39813,127.0.98.61:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:43.878611 6947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:43.878813 6947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:43.878834 6947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:43.880020 6947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:43.880065 6947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:43.880079 6947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:43.880096 6947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:43.880110 6947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:43.881572 6947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39813,127.0.98.61:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
W20260929 18:58:43.881711 6947 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39813,127.0.98.61:39317'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:58:43.881855 6947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:43.882085 6947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:43.886940 6955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.886978 6953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:43.886979 6952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:43.887137 6947 server_base.cc:1034] running on GCE node
I20260929 18:58:43.887507 6947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:43.887815 6947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:43.888995 6947 hybrid_clock.cc:648] HybridClock initialized: now 1790708323888987 us; error 40 us; skew 500 ppm
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:43.891803 6947 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 18:58:43.892248 6947 webserver.cc:460] Webserver started at http://127.0.98.61:33859/ using document root <none> and password file <none>
I20260929 18:58:43.892524 6947 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.892593 6947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.892745 6947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:43.893962 6947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/instance:
uuid: "2f603fad9fd24ac282b9c3df456c8905"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "a43e2b0a9b6e1440fa505d9484f1d677"
server_key_iv: "709fa14a3afa13fee29c697e48dbbac0"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.894332 6947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal/instance:
uuid: "2f603fad9fd24ac282b9c3df456c8905"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "a43e2b0a9b6e1440fa505d9484f1d677"
server_key_iv: "709fa14a3afa13fee29c697e48dbbac0"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.895735 6947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:43.896518 6962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.896677 6947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:43.896775 6947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "2f603fad9fd24ac282b9c3df456c8905"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "a43e2b0a9b6e1440fa505d9484f1d677"
server_key_iv: "709fa14a3afa13fee29c697e48dbbac0"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.896850 6947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.918694 6947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:43.920644 6947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:43.920799 6947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:43.924881 6947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39317
I20260929 18:58:43.924929 7014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39317 every 8 connection(s)
I20260929 18:58:43.925293 6947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:58:43.926216 7015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:43.927736 7015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:43.933089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 6947
I20260929 18:58:43.936652 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.928425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.98.62:39813)
0929 18:58:43.928501 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.928503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.928658 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:58:43.929118 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.929119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.929129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.929172 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.929735 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.930224 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.931224 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.931481 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.931539 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.933765 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.933766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.933766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.933773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.935195 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.935606 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.935608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.935608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.935673 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.935895 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.935896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.935896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.936089 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.936163 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.936519 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.936585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:58:43.936851 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.928368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57636)
0929 18:58:43.928520 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.928577 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.929013 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.929228 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.929231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.929463 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.929655 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.929658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.930298 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.930298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.931137 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.931139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.931568 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.931569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.931669 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.932090 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.932100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.933534 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.935299 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.935304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.935306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.935530 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.935741 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.935741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.935741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.935833 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.936154 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.936158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.936356 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.936463 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.936609 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260929 18:58:43.937542 7015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:58:43.942588 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.933629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.0.98.61:39317)
0929 18:58:43.933781 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.933786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.933810 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:58:43.933866 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.933869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.933881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.933988 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.934550 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.935217 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.936198 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.936653 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.936737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.939600 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.939607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.939607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.939614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.941354 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.941744 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.941746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.941746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.941822 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.942055 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.942056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.942056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.942193 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.942358 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.942406 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.942480 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260929 18:58:43.945382 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.937796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37838)
0929 18:58:43.937845 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.937855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.937901 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.938088 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.938088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.938226 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.938280 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.938282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.939235 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.939236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.939955 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.939957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.940396 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.940397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.940473 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.940738 (+ 265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.940742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.942116 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.944402 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.944403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.944403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.944561 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.944767 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.944767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.944768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.944821 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.945081 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.945082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.945117 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.945185 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.945278 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:58:43.945467 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.937830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.98.61:39317)
0929 18:58:43.937976 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.937978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.937984 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:43.938000 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.938000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.938009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.938087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.938364 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.939112 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.940058 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.940295 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.940381 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.942197 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.942198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.942198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.942205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.944301 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.944642 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.944643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.944644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.944702 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.944892 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.944893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.944893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.945013 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.945089 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.945213 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.945328 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260929 18:58:43.946676 7015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Bootstrap starting.
I20260929 18:58:43.947233 7015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:43.947530 7015 log.cc:824] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:43.948210 7015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: No bootstrap required, opened a new log
I20260929 18:58:43.949471 7015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } }
I20260929 18:58:43.949587 7015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:43.949609 7015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f603fad9fd24ac282b9c3df456c8905, State: Initialized, Role: FOLLOWER
I20260929 18:58:43.949699 7015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } }
I20260929 18:58:43.949923 7015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:43.949913 7022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } } }
I20260929 18:58:43.950027 7022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:58:43.951644 7033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:58:43.951700 7033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:58:43.953174 6947 master_runner.cc:428] Detected that this master 2f603fad9fd24ac282b9c3df456c8905 is joining an existing cluster
I20260929 18:58:43.953223 6947 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39317
I20260929 18:58:43.961462 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.954442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.98.61:39317)
0929 18:58:43.954473 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.954475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.954529 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:58:43.954589 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.954590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.954598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.954640 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.954994 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.955762 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.956702 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.956944 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.957017 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.958212 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.958213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.958214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.958221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.960078 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960571 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.960572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.960573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.960640 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960872 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.960873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.960873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.960997 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.961077 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.961221 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.961308 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 18:58:43.961495 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.954178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.0.98.62:39813)
0929 18:58:43.954226 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.954228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.954360 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:58:43.954411 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.954412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.954423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.954491 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.954940 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.955739 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.956934 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.957203 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.957302 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.958347 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.958348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.958348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.958356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.960164 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960665 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.960667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.960667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.960724 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960918 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.960919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.960919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.961068 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.961152 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.961317 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.961394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:43.961727 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.954309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37844)
0929 18:58:43.954488 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.954496 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.954541 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.954658 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.954659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.954839 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.954911 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.954912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.955849 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.955849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.956570 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.956573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.957023 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.957023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.957094 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.957642 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.957647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.958098 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.960179 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.960181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.960491 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.960707 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.960708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.960765 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.961079 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.961080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.961122 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.961199 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.961657 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":356,"thread_start_us":58,"threads_started":1}
I20260929 18:58:43.962065 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.954126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57646)
0929 18:58:43.954282 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.954293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.954353 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.954539 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.954539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.954779 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.954848 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.954850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.955855 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.955856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.956842 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.956844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.957300 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.957301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.957362 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.957845 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.957850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.958260 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.960236 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.960237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.960510 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.960781 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.960781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.960781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.960854 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.961142 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.961143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.961188 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.961283 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.961373 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
Sep 29 18:58:43 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:58:43.971697 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.964362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37850)
0929 18:58:43.964541 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.964594 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.964730 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.964896 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.964899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.965142 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.965187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.965188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.966053 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.966054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.966654 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.966656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.967100 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.967101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.967152 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.967609 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.967613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.968917 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:43.970547 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.970553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.970555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.970767 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.970992 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:43.970993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:43.970993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:43.971116 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:43.971393 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:43.971399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:43.971442 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:58:43.971505 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:58:43.971604 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260929 18:58:43.971700 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.964425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.0.98.61:39317)
0929 18:58:43.964478 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.964480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.964685 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:58:43.964814 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.964815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.964825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.964892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.965256 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.965957 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.966733 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.967020 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.967097 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.969012 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.969014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.969014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.969020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:43.970441 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.970843 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.970844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.970844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.970900 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:43.971188 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:43.971189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:43.971189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:43.971329 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:43.971389 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:43.971546 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:58:43.971615 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:58:43.972304 6482 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39317: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39317
}
attrs {
promote: true
}
}
dest_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db"
cas_config_opid_index: -1
I20260929 18:58:43.972626 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } }
I20260929 18:58:43.973138 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } }
I20260929 18:58:43.973471 6989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:43.974115 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39317
I20260929 18:58:43.974251 6989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:58:43.974351 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } }
I20260929 18:58:43.974421 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:43.974493 6947 master.cc:537] Master@127.0.98.61:39317 shutting down...
I20260929 18:58:43.974727 7042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:58:43.975545 6468 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 18:58:43.975925 6468 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db -> Peer 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61:39317): Couldn't send request to peer 2f603fad9fd24ac282b9c3df456c8905. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:58:43.977351 6947 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:43.977434 6947 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:43.977481 6947 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: stopping tablet replica
I20260929 18:58:43.990422 6947 master.cc:552] Master@127.0.98.61:39317 shutdown complete.
I20260929 18:58:43.990494 6947 master_runner.cc:305] Clearing existing system tablet
I20260929 18:58:43.990622 6947 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:43.990689 6947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:43.991319 6947 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260929 18:58:43.991865 7047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:43.991997 6947 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:43.992057 6947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:43.992086 6947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "2f603fad9fd24ac282b9c3df456c8905"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "a43e2b0a9b6e1440fa505d9484f1d677"
server_key_iv: "709fa14a3afa13fee29c697e48dbbac0"
server_key_version: "encryptionkey@0"
I20260929 18:58:43.992688 6947 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:43.994086 6947 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:58:43.994218 6947 log.cc:1196] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:58:43.994421 6947 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Deleting consensus metadata
I20260929 18:58:43.994556 6947 master_runner.cc:315] Copying system tablet from 127.0.98.62:39813
I20260929 18:58:43.995105 6947 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39813
I20260929 18:58:44.001637 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.995471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57660)
0929 18:58:43.995568 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:43.995578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:43.995653 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:43.995788 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:43.995788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:43.995936 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:43.995980 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.995982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.996893 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.996893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.997475 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:43.997477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.997920 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.997921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:43.997971 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.998595 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:43.998599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:43.998886 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.000706 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.000706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.000707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.000863 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.001065 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.001065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.001065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.001114 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.001404 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.001404 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.001439 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.001488 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.001560 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":63,"threads_started":1}
I20260929 18:58:44.001675 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:43.995532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.0.98.62:39813)
0929 18:58:43.995584 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:43.995587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:43.995641 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:58:43.995714 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:43.995716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:43.995726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:43.995785 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:43.996053 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.996812 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.997552 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:43.997846 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:43.997913 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:43.998971 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:43.998972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:43.998972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:43.998980 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.000629 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.000933 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.000934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.000935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.000994 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.001214 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.001215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.001215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.001342 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.001410 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.001513 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.001585 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:58:44.002293 6503 tablet_copy_service.cc:140] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2f603fad9fd24ac282b9c3df456c8905 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57660)
I20260929 18:58:44.002391 6503 tablet_copy_service.cc:161] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2f603fad9fd24ac282b9c3df456c8905 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57660: session id = 2f603fad9fd24ac282b9c3df456c8905-00000000000000000000000000000000
I20260929 18:58:44.002982 6503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:58:44.003602 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:58:44.004436 6947 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: tablet copy: Starting download of 0 data blocks...
I20260929 18:58:44.004587 6947 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: tablet copy: Starting download of 1 WAL segments...
I20260929 18:58:44.005478 6947 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:58:44.006209 6503 tablet_copy_service.cc:342] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Request end of tablet copy session 2f603fad9fd24ac282b9c3df456c8905-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57660
I20260929 18:58:44.006273 6503 tablet_copy_service.cc:433] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: ending tablet copy session 2f603fad9fd24ac282b9c3df456c8905-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2f603fad9fd24ac282b9c3df456c8905
I20260929 18:58:44.008524 6947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.009680 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.009801 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.009831 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.010001 6947 server_base.cc:1034] running on GCE node
I20260929 18:58:44.010118 6947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.010243 6947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.011325 6947 hybrid_clock.cc:648] HybridClock initialized: now 1790708324011318 us; error 22 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.012596 6947 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 18:58:44.012877 6947 webserver.cc:460] Webserver started at http://127.0.98.61:39477/ using document root <none> and password file <none>
I20260929 18:58:44.013007 6947 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.013048 6947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.013711 6947 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:44.014219 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.014348 6947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.014393 6947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "2f603fad9fd24ac282b9c3df456c8905"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "a43e2b0a9b6e1440fa505d9484f1d677"
server_key_iv: "709fa14a3afa13fee29c697e48dbbac0"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.014444 6947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.021720 6947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.023214 6947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.026240 6947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39317
I20260929 18:58:44.026577 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39317 every 8 connection(s)
I20260929 18:58:44.026671 6947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:58:44.027562 7117 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:58:44.028033 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Bootstrap starting.
I20260929 18:58:44.030812 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:58:44.031032 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: Bootstrap complete.
I20260929 18:58:44.031221 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } }
I20260929 18:58:44.031322 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f603fad9fd24ac282b9c3df456c8905, State: Initialized, Role: LEARNER
I20260929 18:58:44.031364 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } }
I20260929 18:58:44.031524 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:44.031523 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } }
I20260929 18:58:44.031632 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:58:44.033030 7131 catalog_manager.cc:1259] Loaded cluster ID: 0a20dd9ee4fd4d9a91ab02a7950853cb
I20260929 18:58:44.033077 7131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: loading cluster ID for follower catalog manager: success
I20260929 18:58:44.034250 7131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: acquiring CA information for follower catalog manager: success
I20260929 18:58:44.034511 7131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:58:44.051465 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.044327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.0.98.61:39317)
0929 18:58:44.044452 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.044454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.044462 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:44.044484 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.044484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.044493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.044571 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.045280 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.045926 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.046976 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.047171 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.047228 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.048607 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.048608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.048609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.048614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.050047 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.050497 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.050499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.050499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.050585 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.050816 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.050817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.050817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.050929 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.051040 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.051172 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.051242 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 18:58:44.078312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36863
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:44.159225 7135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:44.159435 7135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:44.159454 7135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:44.160622 7135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:44.160661 7135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:44.160676 7135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:44.160687 7135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:44.160699 7135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:44.162176 7135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:44.162431 7135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:44.162786 7135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.167925 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.167917 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.168149 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.168165 7135 server_base.cc:1034] running on GCE node
I20260929 18:58:44.168483 7135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.168707 7135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.169862 7135 hybrid_clock.cc:648] HybridClock initialized: now 1790708324169847 us; error 29 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.172477 7135 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 18:58:44.172928 7135 webserver.cc:460] Webserver started at http://127.0.98.60:45193/ using document root <none> and password file <none>
I20260929 18:58:44.173152 7135 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.173197 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.173336 7135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:44.174310 7135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/instance:
uuid: "d724c2210e034e1795ab197c8ad0ed0d"
format_stamp: "Formatted at 2026-09-29 18:58:44 on dist-test-slave-gfsd"
server_key: "c310222cdecf23f2ec35bc8d4b1d8388"
server_key_iv: "ecf63522c8bfa76383d298a990193076"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.174662 7135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal/instance:
uuid: "d724c2210e034e1795ab197c8ad0ed0d"
format_stamp: "Formatted at 2026-09-29 18:58:44 on dist-test-slave-gfsd"
server_key: "c310222cdecf23f2ec35bc8d4b1d8388"
server_key_iv: "ecf63522c8bfa76383d298a990193076"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.176007 7135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:44.176744 7150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.176971 7135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:44.177044 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "d724c2210e034e1795ab197c8ad0ed0d"
format_stamp: "Formatted at 2026-09-29 18:58:44 on dist-test-slave-gfsd"
server_key: "c310222cdecf23f2ec35bc8d4b1d8388"
server_key_iv: "ecf63522c8bfa76383d298a990193076"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.177088 7135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.185680 7135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.187927 7135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:44.188081 7135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.192801 7135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36863
I20260929 18:58:44.192830 7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36863 every 8 connection(s)
I20260929 18:58:44.193197 7135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:58:44.194098 7203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:44.195726 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:44.203253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7135
I20260929 18:58:44.205077 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.196552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.98.62:39813)
0929 18:58:44.196633 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.196634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.196664 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:58:44.197115 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.197116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.197125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.197187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.197763 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.198307 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.199384 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.199610 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.199689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.201858 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.201860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.201860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.201867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.203483 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.203943 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.203944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.203945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.204029 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.204336 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.204337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.204338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.204494 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.204575 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.204884 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.204975 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:58:44.205209 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.196379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57664)
0929 18:58:44.196535 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.196605 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.197031 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.197238 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.197241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.197470 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.197668 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.197677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.198431 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.198432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.199244 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.199246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.199707 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.199708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.199837 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.200333 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.200341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.201759 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.203598 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.203605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.203607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.203854 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.204098 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.204099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.204099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.204247 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.204596 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.204601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.204741 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.204836 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.204961 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260929 18:58:44.205687 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:58:44.213419 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.205943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37876)
0929 18:58:44.206050 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.206061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.206105 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.206250 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.206250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.206353 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.206392 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.206393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.207222 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.207223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.208012 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.208015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.208519 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.208520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.208565 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.208964 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.208968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.210399 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.212206 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.212208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.212413 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.212604 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.212605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.212682 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.212992 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.212993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.213048 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.213187 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.213296 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 18:58:44.213547 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.204046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.0.98.60:36863)
0929 18:58:44.204217 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.204221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.204237 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:58:44.204257 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.204259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.204273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.204347 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.205034 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.206013 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.207002 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.207400 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.207471 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.209997 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.210003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.210003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.210009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.211960 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212365 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.212367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.212368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.212417 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212689 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.212689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.212690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.212868 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.213071 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.213231 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.213353 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260929 18:58:44.213573 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.206004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.98.61:39317)
0929 18:58:44.206044 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.206046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.206093 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:58:44.206171 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.206172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.206183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.206242 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.206464 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.207139 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.208086 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.208439 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.208523 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.210488 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.210490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.210490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.210497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.212136 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212478 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.212479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.212479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.212553 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.212748 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.212749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.212749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.212868 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.212977 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.213238 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.213321 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:44.214006 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:58:44.220055 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.214282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53148)
0929 18:58:44.214364 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.214373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.214399 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.214497 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.214498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.214662 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.214703 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.214704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.215411 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.215412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.215892 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.215893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.216257 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.216258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.216294 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.216653 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.216657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.217889 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.219228 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.219229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.219230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.219427 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.219608 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.219609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.219609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.219661 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.219830 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.219830 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.219861 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.219909 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.219977 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 18:58:44.220104 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.214327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.98.60:36863)
0929 18:58:44.214384 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.214385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.214417 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:58:44.214449 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.214450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.214457 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.214496 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.214768 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.215347 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.215981 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.216186 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.216248 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.217961 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.217962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.217962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.217968 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.219133 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.219488 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.219489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.219490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.219535 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.219700 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.219701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.219701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.219783 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.219826 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.219924 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.220018 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":39}
I20260929 18:58:44.221601 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Bootstrap starting.
I20260929 18:58:44.222374 7203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:44.222616 7203 log.cc:824] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:44.223433 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: No bootstrap required, opened a new log
I20260929 18:58:44.225274 7203 raft_consensus.cc:348] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } }
I20260929 18:58:44.225399 7203 raft_consensus.cc:374] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:44.225420 7203 raft_consensus.cc:729] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d724c2210e034e1795ab197c8ad0ed0d, State: Initialized, Role: FOLLOWER
I20260929 18:58:44.225502 7203 consensus_queue.cc:260] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } }
I20260929 18:58:44.225744 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } } }
I20260929 18:58:44.225786 7203 sys_catalog.cc:564] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:44.225831 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:58:44.227612 7219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:58:44.227747 7219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:58:44.229584 7135 master_runner.cc:428] Detected that this master d724c2210e034e1795ab197c8ad0ed0d is joining an existing cluster
I20260929 18:58:44.229633 7135 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36863
I20260929 18:58:44.238147 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.231374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.98.62:39813)
0929 18:58:44.231418 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.231420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.231452 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:58:44.231539 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.231541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.231552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.231603 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.232121 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.232911 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.234108 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.234390 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.234430 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.235532 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.235533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.235534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.235540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.236859 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.237304 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.237305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.237305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.237360 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.237590 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.237591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.237592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.237728 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.237803 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.237962 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.238034 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:58:44.238155 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.231014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57670)
0929 18:58:44.231398 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.231408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.231466 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.231661 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.231662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.231959 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.232032 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.232033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.233061 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.233062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.234009 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.234011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.234509 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.234510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.234571 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.234970 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.234976 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.235449 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.236952 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.236952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.236953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.237221 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.237433 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.237434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.237434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.237494 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.237803 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.237804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.237841 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.237928 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.238027 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260929 18:58:44.240353 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.231250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.98.61:39317)
0929 18:58:44.231315 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.231318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.231628 (+ 310us) server_negotiation.cc:407] Connection header received
0929 18:58:44.231740 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.231740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.231750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.231783 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.232237 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.232983 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.234108 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.235003 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.235048 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.236691 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.236692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.236693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.236699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.238558 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.238952 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.238954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.238954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.239019 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.239654 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.239655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.239655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.239790 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.239875 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.240182 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.240264 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:44.240384 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.231205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37892)
0929 18:58:44.231578 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.231587 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.231667 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.231844 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.231844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.232095 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.232155 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.232157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.233097 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.233097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.234009 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.234011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.235126 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.235126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.235210 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.235646 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.235651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.236593 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.238661 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.238662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.238662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.238877 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.239099 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.239100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.239100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.239582 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.240020 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.240021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.240055 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.240144 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.240253 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:58:44.262789 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.231036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.0.98.60:36863)
0929 18:58:44.231072 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.231074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.231220 (+ 146us) server_negotiation.cc:407] Connection header received
0929 18:58:44.231410 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.231411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.231420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.231447 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.231884 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.232609 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.233635 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.233905 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.233944 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.259819 (+ 25875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.259821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.259822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.259830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.261485 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.261861 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.261862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.261863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.261924 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.262118 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.262119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.262120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.262281 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.262351 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.262540 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.262683 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":18}
I20260929 18:58:44.262796 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.240348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53170)
0929 18:58:44.240428 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.240438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.240473 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.240855 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.240856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.241005 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.241045 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.241047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.242008 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.242009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.242521 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.242523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.242904 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.242904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.242967 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.243377 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.243382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.244628 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.261652 (+ 17024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.261653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.261654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.261887 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.262065 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.262066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.262066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.262139 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.262414 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.262415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.262466 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.262536 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.262678 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 18:58:44.262807 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.230992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53154)
0929 18:58:44.231158 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.231169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.231252 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.231499 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.231499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.231726 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.231807 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.231809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.232707 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.232707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.233532 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.233535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.234026 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.234026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.234078 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.234505 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.234510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.259664 (+ 25154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.261576 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.261577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.261577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.261786 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.261988 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.261988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.261989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.262045 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.262379 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.262379 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.262430 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.262506 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.262625 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":122,"threads_started":1}
I20260929 18:58:44.263114 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.240540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.98.60:36863)
0929 18:58:44.240684 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.240686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.240695 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:44.240733 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.240733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.240746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.241109 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.241157 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.241934 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.242615 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.242833 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.242906 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.244713 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.244714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.244715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.244720 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.261527 (+ 16807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.261954 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.261955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.261955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.262008 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.262208 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.262209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.262209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.262351 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.262416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.262924 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.263008 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":101,"threads_started":1}
I20260929 18:58:44.264098 6482 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36863
}
attrs {
promote: true
}
}
dest_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db"
cas_config_opid_index: 5
I20260929 18:58:44.264361 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } }
I20260929 18:58:44.264873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6545
I20260929 18:58:44.264889 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.265733 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36863
I20260929 18:58:44.265810 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.265875 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.267776 7135 master.cc:537] Master@127.0.98.60:36863 shutting down...
I20260929 18:58:44.267975 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:44.268850 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:58:44.269157 7042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:58:44.269774 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db -> Peer d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60:36863): Couldn't send request to peer d724c2210e034e1795ab197c8ad0ed0d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:58:44.272171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33879
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39813
--webserver_port=33863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:58:44.273399 7135 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:44.273491 7135 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:44.273535 7135 tablet_replica.cc:330] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: stopping tablet replica
I20260929 18:58:44.273850 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.264848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37900)
0929 18:58:44.269851 (+ 5003us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.269862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.269907 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.270124 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.270125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.270126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.270169 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.270170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.271084 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.271085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.272316 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.272318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.273605 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.273606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.273674 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.273696 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.273746 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.273783 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4987}
I20260929 18:58:44.274026 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.264905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.98.61:39317)
0929 18:58:44.264991 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.264993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.269963 (+ 4970us) server_negotiation.cc:407] Connection header received
0929 18:58:44.269990 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.269992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.270014 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.270068 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.270236 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.270988 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.272818 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.273393 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.273464 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.273535 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.273934 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.273970 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:58:44.274276 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 LEARNER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:58:44.274614 7042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:58:44.275202 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.276325 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.276414 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:58:44.276527 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6b66361c5b4b6c9f6bfbd1db7d15db. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.276643 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:58:44.276851 7042 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: attempting to promote NON_VOTER 2f603fad9fd24ac282b9c3df456c8905 to VOTER
I20260929 18:58:44.277035 7042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } }
I20260929 18:58:44.277540 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.277447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53176)
0929 18:58:44.277477 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.277515 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 18:58:44.277532 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 LEARNER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:58:44.277698 7232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 18:58:44.277885 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.277800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53182)
0929 18:58:44.277835 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.277865 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 18:58:44.277971 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db -> Peer d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60:36863): Couldn't send request to peer d724c2210e034e1795ab197c8ad0ed0d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:58:44.278358 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.279199 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.279269 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.279237 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.279340 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.279430 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.280172 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.280251 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:58:44.287329 7135 master.cc:552] Master@127.0.98.60:36863 shutdown complete.
I20260929 18:58:44.287386 7135 master_runner.cc:305] Clearing existing system tablet
I20260929 18:58:44.287529 7135 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.287585 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.288420 7135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:44.289002 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.289156 7135 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.289227 7135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:44.289256 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "d724c2210e034e1795ab197c8ad0ed0d"
format_stamp: "Formatted at 2026-09-29 18:58:44 on dist-test-slave-gfsd"
server_key: "c310222cdecf23f2ec35bc8d4b1d8388"
server_key_iv: "ecf63522c8bfa76383d298a990193076"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.290225 7135 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:44.291936 7135 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:58:44.292018 7135 log.cc:1196] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:58:44.292240 7135 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Deleting consensus metadata
I20260929 18:58:44.292524 7135 master_runner.cc:315] Copying system tablet from 127.0.98.62:39813
I20260929 18:58:44.293211 7135 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39813
I20260929 18:58:44.300884 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.293515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:57676)
0929 18:58:44.293630 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.293650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.293705 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.293839 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.293840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.294033 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.294082 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.294083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.295016 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.295017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.295920 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.295923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.296397 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.296398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.296455 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.297392 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.297398 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.297787 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.299738 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.299739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.299740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.299993 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.300214 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.300214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.300215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.300283 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.300553 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.300554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.300601 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.300682 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.300782 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260929 18:58:44.300884 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.293582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.0.98.62:39813)
0929 18:58:44.293627 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.293629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.293691 (+ 62us) server_negotiation.cc:407] Connection header received
0929 18:58:44.293759 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.293760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.293769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.293837 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.294153 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.294932 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.296016 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.296317 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.296391 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.297887 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.297889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.297889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.297897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.299652 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.300075 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.300076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.300077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.300141 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.300352 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.300353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.300353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.300483 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.300553 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.300725 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.300800 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:44.301431 6503 tablet_copy_service.cc:140] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d724c2210e034e1795ab197c8ad0ed0d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57676)
I20260929 18:58:44.301486 6503 tablet_copy_service.cc:161] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d724c2210e034e1795ab197c8ad0ed0d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57676: session id = d724c2210e034e1795ab197c8ad0ed0d-00000000000000000000000000000000
I20260929 18:58:44.302059 6503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:58:44.302529 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:58:44.303773 7135 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: tablet copy: Starting download of 0 data blocks...
I20260929 18:58:44.303916 7135 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: tablet copy: Starting download of 1 WAL segments...
I20260929 18:58:44.304626 7135 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:58:44.305527 6503 tablet_copy_service.cc:342] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Request end of tablet copy session d724c2210e034e1795ab197c8ad0ed0d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57676
I20260929 18:58:44.305606 6503 tablet_copy_service.cc:433] P 5a6b66361c5b4b6c9f6bfbd1db7d15db: ending tablet copy session d724c2210e034e1795ab197c8ad0ed0d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d724c2210e034e1795ab197c8ad0ed0d
I20260929 18:58:44.308385 7135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.309630 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.309649 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.310024 7135 server_base.cc:1034] running on GCE node
W20260929 18:58:44.313453 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.313603 7135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.313766 7135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.314894 7135 hybrid_clock.cc:648] HybridClock initialized: now 1790708324314886 us; error 28 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.316509 7135 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 18:58:44.316784 7135 webserver.cc:460] Webserver started at http://127.0.98.60:43149/ using document root <none> and password file <none>
I20260929 18:58:44.316919 7135 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.316962 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.317725 7135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:44.318323 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.318498 7135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:44.318550 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "d724c2210e034e1795ab197c8ad0ed0d"
format_stamp: "Formatted at 2026-09-29 18:58:44 on dist-test-slave-gfsd"
server_key: "c310222cdecf23f2ec35bc8d4b1d8388"
server_key_iv: "ecf63522c8bfa76383d298a990193076"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.318608 7135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.338713 7135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.340620 7135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.343689 7135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36863
I20260929 18:58:44.343789 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36863 every 8 connection(s)
I20260929 18:58:44.344003 7135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:58:44.344918 7304 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:58:44.345387 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Bootstrap starting.
I20260929 18:58:44.348315 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:58:44.348544 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Bootstrap complete.
I20260929 18:58:44.348742 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } }
I20260929 18:58:44.348829 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d724c2210e034e1795ab197c8ad0ed0d, State: Initialized, Role: LEARNER
I20260929 18:58:44.348865 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } }
I20260929 18:58:44.349032 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:44.350344 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.350522 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:58:44.350862 7318 catalog_manager.cc:1259] Loaded cluster ID: 0a20dd9ee4fd4d9a91ab02a7950853cb
I20260929 18:58:44.350948 7318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: loading cluster ID for follower catalog manager: success
I20260929 18:58:44.352062 7318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: acquiring CA information for follower catalog manager: success
I20260929 18:58:44.352355 7318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:58:44.382185 7229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:44.382406 7229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:44.382445 7229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:44.383646 7229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:44.383708 7229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:44.383751 7229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:44.385182 7229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33863
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:44.385430 7229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:44.385649 7229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.390735 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.390735 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.390841 7229 server_base.cc:1034] running on GCE node
W20260929 18:58:44.390774 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.391194 7229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.391497 7229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.392663 7229 hybrid_clock.cc:648] HybridClock initialized: now 1790708324392643 us; error 39 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.395262 7229 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:58:44.395677 7229 webserver.cc:460] Webserver started at http://127.0.98.1:33863/ using document root <none> and password file <none>
I20260929 18:58:44.395848 7229 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.395897 7229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.397248 7229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:44.397817 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.398007 7229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:44.398124 7229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "4285b7f578a0453d9420b21e9346972a"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "5e15a86752796a91d22d5f3b2cb11925"
server_key_iv: "9a602bfe5b36909507afd619395e2f66"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.398445 7229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.407948 7229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.410172 7229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:44.410326 7229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.410547 7229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:44.410838 7229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:44.410890 7229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.410929 7229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:44.410957 7229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.417548 7229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33879
I20260929 18:58:44.417639 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33879 every 8 connection(s)
I20260929 18:58:44.418043 7229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:44.427930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7229
I20260929 18:58:44.428088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6679
I20260929 18:58:44.428751 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.419125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59575 (local address 127.0.98.61:39317)
0929 18:58:44.419177 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.419180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.419312 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:58:44.419770 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.419771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.419783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.419831 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.420684 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421443 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422548 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422815 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422905 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.425431 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.425432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.425433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.425439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.427196 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427617 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.427618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.427618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.427675 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427965 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.427966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.427967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.428122 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.428182 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.428516 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.428599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:44.429296 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.419078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.98.1:59575)
0929 18:58:44.419225 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.419245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.419679 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.420286 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.420287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.420446 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.420571 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.420576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421531 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.421531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.422470 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.422471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422911 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.422977 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.423497 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.423502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.425316 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.427272 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427301 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.427303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.427517 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.427746 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.427748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.427897 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.428209 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.428215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.428373 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.428839 (+ 466us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.429024 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":44,"thread_start_us":59,"threads_started":1}
I20260929 18:58:44.429529 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.418803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:57011)
0929 18:58:44.419210 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.419220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.419673 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.419998 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.420003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.420405 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.420571 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.420576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421530 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.421531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.422466 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.422469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422966 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.423030 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.423654 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.423658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.425659 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.427581 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.427582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.427798 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.428042 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.428042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.428043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.428099 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.429156 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.429157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.429207 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.429298 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.429409 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":69,"thread_start_us":84,"threads_started":1}
I20260929 18:58:44.429819 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.418825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57011 (local address 127.0.98.62:39813)
0929 18:58:44.419020 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.419022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.419328 (+ 306us) server_negotiation.cc:407] Connection header received
0929 18:58:44.419891 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.419892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.419901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.419941 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.420686 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421432 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422573 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422847 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422913 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.425800 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.425801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.425801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.425839 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.427508 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.427896 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.427897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.427898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.427956 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.428974 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.428975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.428975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.429080 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.429163 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.429323 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.429504 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 18:58:44.429814 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39317
I20260929 18:58:44.429908 7449 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.430123 7449 heartbeater.cc:511] Master 127.0.98.61:39317 requested a full tablet report, sending...
I20260929 18:58:44.430186 7448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:44.430243 7448 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.430320 7448 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:44.430576 6482 ts_manager.cc:194] Re-registered known tserver with Master: 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879)
I20260929 18:58:44.430620 7080 ts_manager.cc:194] Registered new tserver with Master: 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879)
I20260929 18:58:44.431129 6482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57011
I20260929 18:58:44.436169 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.418803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.98.1:48905)
0929 18:58:44.418952 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.418993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.419667 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.420083 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.420083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.420370 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.420569 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.420580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421441 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.421442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.422223 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.422225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422651 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.422798 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.423333 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.423342 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.426022 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.434826 (+ 8804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.434827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.434827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.434990 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.435194 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.435195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.435195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.435245 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.435563 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.435564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.435598 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.435994 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.436090 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260929 18:58:44.436277 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.418849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48905 (local address 127.0.98.60:36863)
0929 18:58:44.418966 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.418968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.419087 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:58:44.419983 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.419984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.419995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.420076 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.420682 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.421338 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422314 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.422576 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.422646 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.426123 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.426124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.426125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.426132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.434703 (+ 8571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.435061 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.435063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.435063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.435128 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.435342 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.435343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.435343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.435486 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.435550 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.436070 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.436168 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260929 18:58:44.436611 7447 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36863
I20260929 18:58:44.436666 7447 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.436743 7447 heartbeater.cc:511] Master 127.0.98.60:36863 requested a full tablet report, sending...
I20260929 18:58:44.437018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34925
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39813
--webserver_port=37995
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:58:44.437094 7268 ts_manager.cc:194] Registered new tserver with Master: 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879)
W20260929 18:58:44.517658 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:44.517848 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:44.517871 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:44.519162 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:44.519204 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:44.519232 7455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:44.520877 7455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37995
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:44.521121 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:44.521301 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.526453 7460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.526548 7455 server_base.cc:1034] running on GCE node
W20260929 18:58:44.526450 7461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.526479 7463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.526954 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.527215 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.528393 7455 hybrid_clock.cc:648] HybridClock initialized: now 1790708324528377 us; error 27 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.530951 7455 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:58:44.531517 7455 webserver.cc:460] Webserver started at http://127.0.98.2:37995/ using document root <none> and password file <none>
I20260929 18:58:44.531746 7455 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.531805 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.533232 7455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:44.533926 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.534142 7455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:44.534237 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "a18822a622364ca796a5484927e6ae22"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "e228b2f0a6ddd66e55bbc879c51b4c9f"
server_key_iv: "89fb7bdb595b46b25c2319acdbbf1f51"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.534583 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.554935 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.557281 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:44.557415 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.557696 7455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:44.558000 7455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:44.558058 7455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.558101 7455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:44.558130 7455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.564936 7455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34925
I20260929 18:58:44.565040 7583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34925 every 8 connection(s)
I20260929 18:58:44.565371 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:44.571786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7455
I20260929 18:58:44.571952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6813
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:58:44.577116 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48967 (local address 127.0.98.61:39317)
0929 18:58:44.566511 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.566513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.566707 (+ 194us) server_negotiation.cc:407] Connection header received
0929 18:58:44.567638 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.567640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.567653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.567709 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.568645 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.569163 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570615 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570911 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570978 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.573561 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.573562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.573563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.573571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.575092 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.575639 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.575641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.575641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.575708 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.576002 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.576004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.576004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.576152 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.576213 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.576908 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.576986 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:44.577595 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.98.2:48967)
0929 18:58:44.566647 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.566656 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.567497 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.567798 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.567802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.568340 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.568540 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.568545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.569273 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.569274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.570521 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.570523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570994 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.571130 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.571745 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.571756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.573467 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.575186 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.575193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.575195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.575536 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.575784 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.575785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.575785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.575919 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.576248 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.576254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.576419 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.577222 (+ 803us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.577404 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":58,"threads_started":1}
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:44.578209 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39317
I20260929 18:58:44.578318 7586 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.578510 7586 heartbeater.cc:511] Master 127.0.98.61:39317 requested a full tablet report, sending...
I20260929 18:58:44.578958 7080 ts_manager.cc:194] Registered new tserver with Master: a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:44.579432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36747
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39813
--webserver_port=40467
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:58:44.579690 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.98.2:44293)
0929 18:58:44.566268 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.566309 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.567494 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.567797 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.567802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.568346 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.568541 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.568547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570866 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.571733 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.571736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.572809 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.572810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.572886 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.573315 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.573320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.574922 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.576743 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.576744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.576745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.576996 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.578046 (+ 1050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.578047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.578048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.578170 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.579271 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.579274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.579340 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.579454 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.579559 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260929 18:58:44.579742 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44293 (local address 127.0.98.60:36863)
0929 18:58:44.566229 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.566232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.566396 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:58:44.567698 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.567699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.567714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.567767 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.570122 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570805 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.571841 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.572183 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.572227 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.575018 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.575021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.575021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.575030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.576664 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.577089 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.577090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.577091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.577152 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.578243 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.578244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.578245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.578390 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.578432 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.579496 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.579608 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:58:44.581202 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34719 (local address 127.0.98.62:39813)
0929 18:58:44.566579 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.566581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.566626 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:58:44.567825 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.567827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.567839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.567896 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.569287 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570011 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570801 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.571111 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.571177 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.578269 (+ 7092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.578271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.578271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.578279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.580005 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.580375 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.580376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.580376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.580424 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.580637 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.580638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.580638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.580791 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.580863 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.581029 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.581112 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:58:44.581269 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.566125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:34719)
0929 18:58:44.566572 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.566583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.567497 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.567911 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.567912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.568349 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.568541 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.568547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.570100 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.570101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.570718 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.570720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.571192 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.571193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.571255 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.575892 (+ 4637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.575897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.578159 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.580082 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.580083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.580083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.580302 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.580487 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.580487 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.580488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.580553 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.580863 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.580864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.580909 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.580993 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.581112 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":12,"thread_start_us":85,"threads_started":1}
I20260929 18:58:44.581823 7584 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36863
I20260929 18:58:44.581861 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:44.581887 7584 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.581930 7585 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.581977 7584 heartbeater.cc:511] Master 127.0.98.60:36863 requested a full tablet report, sending...
I20260929 18:58:44.582005 7585 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:44.582252 6482 ts_manager.cc:194] Re-registered known tserver with Master: a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:44.582271 7268 ts_manager.cc:194] Registered new tserver with Master: a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:44.582782 6482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34719
W20260929 18:58:44.662849 7591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:44.663079 7591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:44.663111 7591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:44.664453 7591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:44.664520 7591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:44.664565 7591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:44.665966 7591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40467
--tserver_master_addrs=127.0.98.62:39813,127.0.98.61:39317,127.0.98.60:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:44.666137 7591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:44.666299 7591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:44.671226 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:44.671226 7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.671355 7591 server_base.cc:1034] running on GCE node
W20260929 18:58:44.671259 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:44.671681 7591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:44.671927 7591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:44.673097 7591 hybrid_clock.cc:648] HybridClock initialized: now 1790708324673069 us; error 39 us; skew 500 ppm
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:44.675767 7591 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:58:44.676203 7591 webserver.cc:460] Webserver started at http://127.0.98.3:40467/ using document root <none> and password file <none>
I20260929 18:58:44.676424 7591 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:44.676503 7591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:44.677809 7591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:44.678354 7606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.678512 7591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.678635 7591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "f9fb93a0c7444beb9f83f0180302db3c"
format_stamp: "Formatted at 2026-09-29 18:58:43 on dist-test-slave-gfsd"
server_key: "c553e8e6bd2f2930d248166a6e85f8c5"
server_key_iv: "5f8550ef16f3b10323fa015ea04ae1c6"
server_key_version: "encryptionkey@0"
I20260929 18:58:44.678961 7591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:44.687505 7591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:44.689179 7591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:44.689327 7591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:44.689530 7591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:44.689823 7591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:44.689877 7591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.689913 7591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:44.689939 7591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:44.695681 7591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36747
I20260929 18:58:44.695781 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36747 every 8 connection(s)
I20260929 18:58:44.696077 7591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:58:44 dist-test-slave-gfsd krb5kdc[6438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:44.705255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7591
I20260929 18:58:44.706679 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.696803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40981 (local address 127.0.98.60:36863)
0929 18:58:44.696852 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.696854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.697272 (+ 418us) server_negotiation.cc:407] Connection header received
0929 18:58:44.698075 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.698076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.698088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.698157 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.698986 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699560 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.700535 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.700733 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.700817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.703255 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.703256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.703256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.703262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.705082 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705593 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.705594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.705595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.705656 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705932 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.705933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.705934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.706083 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.706134 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.706503 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.706589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:44.706701 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.697236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46843 (local address 127.0.98.61:39317)
0929 18:58:44.697323 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.697325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.698195 (+ 870us) server_negotiation.cc:407] Connection header received
0929 18:58:44.698305 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.698306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.698318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.698382 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.699007 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699784 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701284 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.701570 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701645 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.703736 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.703737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.703738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.703746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.705534 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705913 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.705914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.705914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.705969 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706188 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.706189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.706189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.706305 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.706346 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.706537 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.706604 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:58:44.707221 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.697072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.98.3:46843)
0929 18:58:44.698140 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.698150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.698229 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.698398 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.698399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.698643 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.698898 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.698904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699868 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.699869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.701207 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.701208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.701638 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.701705 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.702124 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.702129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.703620 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.705614 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.705615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.705828 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.706050 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.706051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.706107 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.706392 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.706393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.706426 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.706800 (+ 374us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.706978 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":16,"thread_start_us":55,"threads_started":1}
I20260929 18:58:44.707221 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.696791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.98.3:40981)
0929 18:58:44.697188 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.697198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.697985 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.698213 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.698217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.698688 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.698897 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.698902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699666 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.699667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.700430 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.700432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.700826 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.700827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.700943 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.701478 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.701487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.703162 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.705189 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.705197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.705499 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.705730 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.705731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.705731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.705856 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.706190 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.706196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.706346 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.706778 (+ 432us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.707021 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":134,"thread_start_us":93,"threads_started":1}
I20260929 18:58:44.707708 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39317
I20260929 18:58:44.707809 7722 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.707921 7720 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36863
I20260929 18:58:44.707967 7720 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.708005 7722 heartbeater.cc:511] Master 127.0.98.61:39317 requested a full tablet report, sending...
I20260929 18:58:44.708035 7720 heartbeater.cc:511] Master 127.0.98.60:36863 requested a full tablet report, sending...
I20260929 18:58:44.708429 7080 ts_manager.cc:194] Registered new tserver with Master: f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3:36747)
I20260929 18:58:44.708429 7268 ts_manager.cc:194] Registered new tserver with Master: f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3:36747)
I20260929 18:58:44.708740 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.696832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49417 (local address 127.0.98.62:39813)
0929 18:58:44.697029 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.697031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.697046 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:58:44.698083 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.698084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.698097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.698149 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.699008 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699769 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701308 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.701614 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701681 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.704283 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.704285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.704285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.704293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.706008 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706397 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.706398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.706398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.706460 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706667 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.706668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.706668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.706805 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.706849 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.708567 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.708650 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260929 18:58:44.709024 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.696791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:49417)
0929 18:58:44.696958 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.696991 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.697961 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.698215 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.698218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.698647 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:44.698898 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.698904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.699868 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.699869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.701202 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.701204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.701736 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.701737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.701796 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.702578 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:44.702582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:44.704194 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:44.706108 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.706109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.706332 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.706532 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:44.706533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:44.706533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:44.706590 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:44.707273 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:44.707274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:44.707461 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.708441 (+ 980us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.708903 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":56,"thread_start_us":96,"threads_started":1}
I20260929 18:58:44.709370 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39813
I20260929 18:58:44.709421 7721 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:44.709473 7721 heartbeater.cc:511] Master 127.0.98.62:39813 requested a full tablet report, sending...
I20260929 18:58:44.709717 6482 ts_manager.cc:194] Re-registered known tserver with Master: f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3:36747)
I20260929 18:58:44.710211 6482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49417
I20260929 18:58:44.716588 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.706911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.60:36863)
0929 18:58:44.706953 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.706955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.707824 (+ 869us) server_negotiation.cc:407] Connection header received
0929 18:58:44.707877 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.707878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.707890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.707946 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.709408 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.710086 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.711199 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.711588 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.711641 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.713178 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.713180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.713180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.713186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.715058 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.715483 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.715484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.715485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.715541 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.715779 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.715780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.715780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.715916 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.716016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.716406 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.716474 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:58:44.716928 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.706582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.0.98.62:39813)
0929 18:58:44.707696 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.707698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.707998 (+ 300us) server_negotiation.cc:407] Connection header received
0929 18:58:44.708019 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.708020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.708032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.708094 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.709129 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.709886 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.710937 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.711156 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.711244 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.713197 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.713198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.713198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.713205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.715477 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.715849 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.715850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.715851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.715900 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.716078 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.716079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.716079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.716217 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.716311 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.716499 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.716586 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":77,"threads_started":1}
I20260929 18:58:44.717072 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.707445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.0.98.61:39317)
0929 18:58:44.707481 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.707483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.708205 (+ 722us) server_negotiation.cc:407] Connection header received
0929 18:58:44.708940 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.708941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.708949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.708976 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.710636 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.711316 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.712305 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.712637 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.712717 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.714181 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.714183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.714183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.714190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.715918 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.716278 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.716279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.716279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.716333 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.716543 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.716544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.716544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.716656 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.716736 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.716926 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.716999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:44.718776 6482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:44.720366 6482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:44.727126 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.723824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33879 (local address 127.0.0.1:56156)
0929 18:58:44.723981 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.723992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.724036 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.724463 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.724464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.724466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.724509 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.724510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.725380 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.725381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.726118 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.726120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.726884 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.726885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.726947 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.726952 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.727012 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.727050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":21,"thread_start_us":117,"threads_started":1}
I20260929 18:58:44.727146 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.724078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.98.1:33879)
0929 18:58:44.724251 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.724256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.724268 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:58:44.724312 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.724314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.724331 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.724413 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.724595 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.725289 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.726221 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.726788 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.726886 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.726961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.727025 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.727062 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260929 18:58:44.727993 7381 tablet_service.cc:1467] Processing CreateTablet for tablet ab34a9e3a56e494c94390d21cce0a868 (DEFAULT_TABLE table=test-table [id=77d97d87f36644d3863c315f9deea4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:44.728155 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.723828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34925 (local address 127.0.0.1:38696)
0929 18:58:44.724273 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.724284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.724362 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.724550 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.724551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.724552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.724584 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.724586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.725411 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.725412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.726910 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.726912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.727901 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.727901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.727980 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.727984 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.728030 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.728067 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":97,"threads_started":1}
I20260929 18:58:44.728363 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.724042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.98.2:34925)
0929 18:58:44.724204 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.724209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.724357 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:58:44.724463 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.724466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.724482 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.724556 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.724629 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.725313 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.726998 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.727762 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.727810 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.727852 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.728246 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.728289 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260929 18:58:44.728374 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34a9e3a56e494c94390d21cce0a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:44.728966 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.723731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36747 (local address 127.0.0.1:57270)
0929 18:58:44.723763 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.723773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.723810 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.725632 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.725633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.725634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.725676 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.725678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.726839 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.726840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.727901 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.727903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.728750 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.728751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.728811 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.728814 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.728872 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.728907 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 18:58:44.728986 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.723827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.98.3:36747)
0929 18:58:44.725469 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.725473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.725481 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:44.725514 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.725516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.725531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.725743 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.725792 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.726764 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.727994 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.728636 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.728747 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.728813 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.728884 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.728915 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":104,"threads_started":1}
I20260929 18:58:44.729561 7654 tablet_service.cc:1467] Processing CreateTablet for tablet ab34a9e3a56e494c94390d21cce0a868 (DEFAULT_TABLE table=test-table [id=77d97d87f36644d3863c315f9deea4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:44.729804 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34a9e3a56e494c94390d21cce0a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:44.731482 7744 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Bootstrap starting.
I20260929 18:58:44.732141 7744 tablet_bootstrap.cc:654] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:44.732048 7518 tablet_service.cc:1467] Processing CreateTablet for tablet ab34a9e3a56e494c94390d21cce0a868 (DEFAULT_TABLE table=test-table [id=77d97d87f36644d3863c315f9deea4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:44.732376 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34a9e3a56e494c94390d21cce0a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:44.732445 7744 log.cc:824] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:44.732849 7745 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Bootstrap starting.
I20260929 18:58:44.733364 7744 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: No bootstrap required, opened a new log
I20260929 18:58:44.733429 7745 tablet_bootstrap.cc:654] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:44.733444 7744 ts_tablet_manager.cc:1397] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:44.733728 7745 log.cc:824] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:44.734947 7745 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: No bootstrap required, opened a new log
I20260929 18:58:44.735015 7745 ts_tablet_manager.cc:1397] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:44.735549 7748 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Bootstrap starting.
I20260929 18:58:44.736111 7748 tablet_bootstrap.cc:654] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:44.736394 7748 log.cc:824] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:44.736450 7744 raft_consensus.cc:348] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.736656 7744 raft_consensus.cc:374] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:44.736706 7744 raft_consensus.cc:729] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4285b7f578a0453d9420b21e9346972a, State: Initialized, Role: FOLLOWER
I20260929 18:58:44.736868 7744 consensus_queue.cc:260] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.737082 7748 tablet_bootstrap.cc:492] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: No bootstrap required, opened a new log
I20260929 18:58:44.737106 7744 ts_tablet_manager.cc:1428] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 18:58:44.737169 7748 ts_tablet_manager.cc:1397] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:44.737412 7448 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
I20260929 18:58:44.737474 7745 raft_consensus.cc:348] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.737669 7745 raft_consensus.cc:374] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:44.737715 7745 raft_consensus.cc:729] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb93a0c7444beb9f83f0180302db3c, State: Initialized, Role: FOLLOWER
I20260929 18:58:44.737839 7745 consensus_queue.cc:260] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.738076 7745 ts_tablet_manager.cc:1428] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:58:44.738353 7721 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
I20260929 18:58:44.740026 7748 raft_consensus.cc:348] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.740172 7748 raft_consensus.cc:374] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:44.740218 7748 raft_consensus.cc:729] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18822a622364ca796a5484927e6ae22, State: Initialized, Role: FOLLOWER
I20260929 18:58:44.740326 7748 consensus_queue.cc:260] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.740574 7748 ts_tablet_manager.cc:1428] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:58:44.740659 7585 heartbeater.cc:503] Master 127.0.98.62:39813 was elected leader, sending a full tablet report...
I20260929 18:58:44.746228 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.743904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53194)
0929 18:58:44.743953 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.743964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.744008 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.744123 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.744123 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.744125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.744173 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.744175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.744855 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.744856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.745541 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.745543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.746050 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.746051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.746092 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.746095 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.746149 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.746178 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:58:44.746240 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.743970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.98.60:36863)
0929 18:58:44.744015 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.744017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.744025 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:44.744044 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.744045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.744060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.744114 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.744215 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.744799 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.745616 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.745981 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.746040 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.746080 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.746145 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.746195 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:58:44.746613 7278 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 18:58:44.747401 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6b66361c5b4b6c9f6bfbd1db7d15db. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.747491 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:58:44.782752 7751 raft_consensus.cc:482] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:44.782883 7751 raft_consensus.cc:504] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.783321 7751 leader_election.cc:288] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879), a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:44.786166 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.783539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52895 (local address 127.0.98.1:33879)
0929 18:58:44.783593 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.783595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.783693 (+ 98us) server_negotiation.cc:407] Connection header received
0929 18:58:44.783756 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.783758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.783771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.783826 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.783987 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.784540 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785441 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.785794 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785900 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.785910 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.786066 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.786104 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:58:44.786161 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.783462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34925 (local address 127.0.98.3:48005)
0929 18:58:44.783519 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.783532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.783612 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.783789 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.783790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.783792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.783866 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.783868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.784553 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.784554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.785316 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.785318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.785867 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.785943 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.785947 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.785998 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.786055 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:58:44.786185 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.783529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48005 (local address 127.0.98.2:34925)
0929 18:58:44.783582 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.783584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.783595 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:44.783704 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.783705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.783718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.783781 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:44.783937 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.784467 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785399 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.785783 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785836 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.785845 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.786020 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.786065 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:58:44.786271 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.783470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33879 (local address 127.0.98.3:52895)
0929 18:58:44.783623 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:44.783634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:44.783689 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:44.783842 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:44.783843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:44.783847 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:44.783916 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.783918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.784623 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.784624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.785324 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:44.785326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.785913 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.785913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:44.785968 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.785971 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:44.786044 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:58:44.786081 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":49}
I20260929 18:58:44.786614 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34a9e3a56e494c94390d21cce0a868" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4285b7f578a0453d9420b21e9346972a" is_pre_election: true
I20260929 18:58:44.786772 7401 raft_consensus.cc:2454] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 0.
I20260929 18:58:44.786907 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34a9e3a56e494c94390d21cce0a868" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18822a622364ca796a5484927e6ae22" is_pre_election: true
I20260929 18:58:44.787031 7538 raft_consensus.cc:2454] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 0.
I20260929 18:58:44.786980 7609 leader_election.cc:302] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4285b7f578a0453d9420b21e9346972a, f9fb93a0c7444beb9f83f0180302db3c; no voters:
I20260929 18:58:44.787194 7751 raft_consensus.cc:2781] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:44.787263 7751 raft_consensus.cc:482] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:44.787329 7751 raft_consensus.cc:3037] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:44.787995 7751 raft_consensus.cc:504] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.788136 7751 leader_election.cc:288] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 election: Requested vote from peers 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879), a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:44.788285 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34a9e3a56e494c94390d21cce0a868" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4285b7f578a0453d9420b21e9346972a"
I20260929 18:58:44.788307 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34a9e3a56e494c94390d21cce0a868" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18822a622364ca796a5484927e6ae22"
I20260929 18:58:44.788362 7401 raft_consensus.cc:3037] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:44.788393 7538 raft_consensus.cc:3037] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:44.789072 7401 raft_consensus.cc:2454] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 1.
I20260929 18:58:44.789088 7538 raft_consensus.cc:2454] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 1.
I20260929 18:58:44.789232 7609 leader_election.cc:302] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4285b7f578a0453d9420b21e9346972a, f9fb93a0c7444beb9f83f0180302db3c; no voters:
I20260929 18:58:44.789326 7751 raft_consensus.cc:2781] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:44.789469 7751 raft_consensus.cc:686] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 1 LEADER]: Becoming Leader. State: Replica: f9fb93a0c7444beb9f83f0180302db3c, State: Running, Role: LEADER
I20260929 18:58:44.789604 7751 consensus_queue.cc:237] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } }
I20260929 18:58:44.790418 6483 catalog_manager.cc:5495] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c reported cstate change: term changed from 0 to 1, leader changed from <none> to f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f9fb93a0c7444beb9f83f0180302db3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:44.791123 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: true } } }
I20260929 18:58:44.791250 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.791952 7759 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: attempting to promote NON_VOTER d724c2210e034e1795ab197c8ad0ed0d to VOTER
I20260929 18:58:44.792106 7759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:44.792577 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 18:58:44.792572 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 LEARNER]: Refusing update from remote peer 5a6b66361c5b4b6c9f6bfbd1db7d15db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 18:58:44.792898 7759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 18:58:44.792999 7232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 18:58:44.793753 7232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.793953 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.794575 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.794660 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.794593 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.794720 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.794688 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.794772 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:44.794858 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.794932 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:58:44.795193 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.795969 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:44.796061 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:58:44.800536 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.798060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.98.3:36747)
0929 18:58:44.798159 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.798161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.798168 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:44.798195 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.798196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.798213 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.798270 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:44.798395 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.798906 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.799757 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.799965 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.800035 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.800200 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:44.800325 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:44.800400 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.800440 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.800459 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:58:44.802325 7401 raft_consensus.cc:1261] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Refusing update from remote peer f9fb93a0c7444beb9f83f0180302db3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:44.802372 7538 raft_consensus.cc:1261] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Refusing update from remote peer f9fb93a0c7444beb9f83f0180302db3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:44.802703 7757 consensus_queue.cc:1035] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:44.802847 7751 consensus_queue.cc:1035] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:44.806289 7765 mvcc.cc:205] Tried to move back new op lower bound from 7334741298387992576 to 7334741298338881536. Current Snapshot: MvccSnapshot[applied={T|T < 7334741298387992576}]
I20260929 18:58:44.808830 6483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:44.808910 6483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:44.810213 6483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6b66361c5b4b6c9f6bfbd1db7d15db: Sending DeleteTablet for 3 replicas of tablet ab34a9e3a56e494c94390d21cce0a868
I20260929 18:58:44.810570 7518 tablet_service.cc:1514] Processing DeleteTablet for tablet ab34a9e3a56e494c94390d21cce0a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38696
I20260929 18:58:44.810657 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet ab34a9e3a56e494c94390d21cce0a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57270
I20260929 18:58:44.810847 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet ab34a9e3a56e494c94390d21cce0a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56156
I20260929 18:58:44.811045 7775 tablet_replica.cc:330] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: stopping tablet replica
I20260929 18:58:44.811194 7775 raft_consensus.cc:2229] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:44.811450 7774 tablet_replica.cc:330] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: stopping tablet replica
I20260929 18:58:44.811542 7774 raft_consensus.cc:2229] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:44.811625 7774 raft_consensus.cc:2258] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:44.811827 7773 tablet_replica.cc:330] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: stopping tablet replica
I20260929 18:58:44.811916 7773 raft_consensus.cc:2229] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:44.812002 7773 raft_consensus.cc:2258] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:44.812608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6451
I20260929 18:58:44.815654 7774 ts_tablet_manager.cc:1905] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:44.817411 7774 ts_tablet_manager.cc:1918] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:44.817488 7774 log.cc:1196] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/ab34a9e3a56e494c94390d21cce0a868
I20260929 18:58:44.817749 7774 ts_tablet_manager.cc:1939] T ab34a9e3a56e494c94390d21cce0a868 P f9fb93a0c7444beb9f83f0180302db3c: Deleting consensus metadata
I20260929 18:58:44.820745 7775 raft_consensus.cc:2258] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:44.821094 7773 ts_tablet_manager.cc:1905] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:44.822031 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.812136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.0.98.60:36863)
0929 18:58:44.812235 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.812237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.812247 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:44.812269 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.812270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.812281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.812352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.813195 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.813953 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.815010 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.815347 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.815392 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.818928 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.818929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.818930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.818936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.820654 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.821204 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.821205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.821206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.821263 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.821466 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.821467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.821467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.821588 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.821665 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.821868 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.821943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:58:44.822954 7773 ts_tablet_manager.cc:1918] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:44.823032 7773 log.cc:1196] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/ab34a9e3a56e494c94390d21cce0a868
I20260929 18:58:44.823307 7775 ts_tablet_manager.cc:1905] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:44.823376 7773 ts_tablet_manager.cc:1939] T ab34a9e3a56e494c94390d21cce0a868 P a18822a622364ca796a5484927e6ae22: Deleting consensus metadata
W20260929 18:58:44.823714 7474 connection.cc:441] server connection from 127.0.0.1:38696 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38696 (request call id 1) could send its response
I20260929 18:58:44.825088 7775 ts_tablet_manager.cc:1918] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:44.825188 7775 log.cc:1196] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/ab34a9e3a56e494c94390d21cce0a868
I20260929 18:58:44.825464 7775 ts_tablet_manager.cc:1939] T ab34a9e3a56e494c94390d21cce0a868 P 4285b7f578a0453d9420b21e9346972a: Deleting consensus metadata
W20260929 18:58:44.825817 7336 connection.cc:441] server connection from 127.0.0.1:56156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56156 (request call id 1) could send its response
W20260929 18:58:44.825878 7609 connection.cc:535] server connection from 127.0.0.1:57270 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:58:44.834326 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.827248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.98.60:36863)
0929 18:58:44.827311 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.827313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.827591 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:58:44.827659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.827660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.827669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.827719 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.828604 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.829095 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.829763 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.829971 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.830043 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.831593 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.831594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.831595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.831601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.833054 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.833567 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.833568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.833569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.833621 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.833822 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.833823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.833823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.833932 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.834035 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.834135 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.834206 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:58:44.835016 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:44.827414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.0.98.61:39317)
0929 18:58:44.827459 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:44.827461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:44.827663 (+ 202us) server_negotiation.cc:407] Connection header received
0929 18:58:44.827757 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:44.827758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:44.827768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:44.827796 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:44.828511 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.829360 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.830394 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:44.830699 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:44.830738 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:44.832190 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:44.832191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:44.832191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:44.832199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:44.833872 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.834259 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.834260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.834261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.834314 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:44.834505 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:44.834506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:44.834506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:44.834631 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:44.834696 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:44.834896 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:58:44.834957 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:58:45.807369 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:45.807092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:53433)
0929 18:58:45.807206 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:45.807307 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260929 18:58:45.807591 7448 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:45.810155 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:45.809857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:60027)
0929 18:58:45.810020 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:45.810103 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20260929 18:58:45.810357 7721 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:45.811619 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:45.811438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:43601)
0929 18:58:45.811528 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:45.811582 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
W20260929 18:58:45.811789 7585 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:46.310617 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a6b66361c5b4b6c9f6bfbd1db7d15db)
I20260929 18:58:46.310751 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:46.310796 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a6b66361c5b4b6c9f6bfbd1db7d15db)
I20260929 18:58:46.310947 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:46.311067 7787 leader_election.cc:288] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61:39317)
I20260929 18:58:46.311313 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60:36863)
I20260929 18:58:46.311590 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58206)
0929 18:58:46.311493 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.311542 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":46,"threads_started":1}
I20260929 18:58:46.311599 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58194)
0929 18:58:46.311459 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.311542 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":34,"threads_started":1}
W20260929 18:58:46.311702 7066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
W20260929 18:58:46.311960 7253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:46.312393 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.312284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58208)
0929 18:58:46.312339 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.312369 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:58:46.312531 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.312443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58210)
0929 18:58:46.312476 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.312505 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 18:58:46.312531 7066 leader_election.cc:334] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
W20260929 18:58:46.312592 7253 leader_election.cc:334] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:46.314424 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39317 (local address 127.0.0.1:37914)
0929 18:58:46.311661 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.311672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:46.311730 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:46.311927 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:46.311928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:46.311930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:46.311991 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:46.311993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.312908 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.312908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:46.313632 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:46.313634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.314203 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.314204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:46.314261 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:46.314264 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:46.314311 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:58:46.314359 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":56,"threads_started":1}
I20260929 18:58:46.314471 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.0.98.61:39317)
0929 18:58:46.311718 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:46.311720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:46.311728 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:46.311818 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:46.311819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:46.311829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:46.311878 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:46.312077 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.312809 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.313737 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.314131 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.314184 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:46.314216 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:46.314331 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:58:46.314379 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":245,"thread_start_us":29,"threads_started":1}
I20260929 18:58:46.314503 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.0.98.60:36863)
0929 18:58:46.311784 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:46.311786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:46.311792 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:58:46.311817 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:46.311818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:46.311829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:46.312051 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:46.312085 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.312782 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.313742 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.314136 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.314222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:46.314232 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:46.314379 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:58:46.314412 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":40,"threads_started":1}
I20260929 18:58:46.314482 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.311449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36863 (local address 127.0.0.1:53226)
0929 18:58:46.311536 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.311546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:46.311638 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:46.311927 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:46.311928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:46.311929 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:46.311986 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:46.311989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.312858 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.312859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:46.313618 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:46.313620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:46.314235 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:46.314237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:46.314285 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:46.314287 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:46.314359 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:58:46.314397 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260929 18:58:46.314659 7091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d724c2210e034e1795ab197c8ad0ed0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f603fad9fd24ac282b9c3df456c8905" is_pre_election: true
I20260929 18:58:46.314803 7091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d724c2210e034e1795ab197c8ad0ed0d in term 1.
I20260929 18:58:46.314960 7253 leader_election.cc:302] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f603fad9fd24ac282b9c3df456c8905, d724c2210e034e1795ab197c8ad0ed0d; no voters: 5a6b66361c5b4b6c9f6bfbd1db7d15db
I20260929 18:58:46.314957 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f603fad9fd24ac282b9c3df456c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d724c2210e034e1795ab197c8ad0ed0d" is_pre_election: true
I20260929 18:58:46.315088 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:58:46.315130 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f603fad9fd24ac282b9c3df456c8905 in term 1.
I20260929 18:58:46.315168 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a6b66361c5b4b6c9f6bfbd1db7d15db)
I20260929 18:58:46.315201 7787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:58:46.315271 7068 leader_election.cc:302] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f603fad9fd24ac282b9c3df456c8905, d724c2210e034e1795ab197c8ad0ed0d; no voters: 5a6b66361c5b4b6c9f6bfbd1db7d15db
I20260929 18:58:46.315395 7788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:58:46.315434 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a6b66361c5b4b6c9f6bfbd1db7d15db)
I20260929 18:58:46.315459 7788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:58:46.315994 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:46.316125 7787 leader_election.cc:288] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 election: Requested vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61:39317)
I20260929 18:58:46.316308 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.316207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58218)
0929 18:58:46.316245 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.316277 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 18:58:46.316334 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:46.316346 7091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d724c2210e034e1795ab197c8ad0ed0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f603fad9fd24ac282b9c3df456c8905"
I20260929 18:58:46.316435 7091 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d724c2210e034e1795ab197c8ad0ed0d in current term 2: Already voted for candidate 2f603fad9fd24ac282b9c3df456c8905 in this term.
I20260929 18:58:46.316466 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 election: Requested vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), d724c2210e034e1795ab197c8ad0ed0d (127.0.98.60:36863)
I20260929 18:58:46.316628 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f603fad9fd24ac282b9c3df456c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d724c2210e034e1795ab197c8ad0ed0d"
I20260929 18:58:46.316713 7278 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f603fad9fd24ac282b9c3df456c8905 in current term 2: Already voted for candidate d724c2210e034e1795ab197c8ad0ed0d in this term.
I20260929 18:58:46.316720 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.316543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58220)
0929 18:58:46.316655 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.316681 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 18:58:46.316809 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.316654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58228)
0929 18:58:46.316760 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.316786 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 18:58:46.316859 7253 leader_election.cc:334] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:46.316885 7253 leader_election.cc:302] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d724c2210e034e1795ab197c8ad0ed0d; no voters: 2f603fad9fd24ac282b9c3df456c8905, 5a6b66361c5b4b6c9f6bfbd1db7d15db
I20260929 18:58:46.316959 7787 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:58:46.317286 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.317209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58242)
0929 18:58:46.317245 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.317268 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 18:58:46.317363 7066 leader_election.cc:334] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:46.317425 7066 leader_election.cc:302] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f603fad9fd24ac282b9c3df456c8905; no voters: 5a6b66361c5b4b6c9f6bfbd1db7d15db, d724c2210e034e1795ab197c8ad0ed0d
I20260929 18:58:46.317494 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:58:46.808329 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.808037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:49823)
0929 18:58:46.808182 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.808268 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260929 18:58:46.810889 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.810676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:43809)
0929 18:58:46.810812 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.810844 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20260929 18:58:46.812216 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:46.812052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:50625)
0929 18:58:46.812155 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:46.812181 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260929 18:58:47.809228 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.808960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:33431)
0929 18:58:47.809117 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.809169 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260929 18:58:47.811458 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.811250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:39271)
0929 18:58:47.811374 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.811409 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260929 18:58:47.812872 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.812665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:56507)
0929 18:58:47.812784 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.812824 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260929 18:58:47.925601 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:47.925709 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:47.925920 7803 leader_election.cc:288] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61:39317)
I20260929 18:58:47.926329 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.926052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58408)
0929 18:58:47.926227 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.926281 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20260929 18:58:47.926311 7091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d724c2210e034e1795ab197c8ad0ed0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f603fad9fd24ac282b9c3df456c8905" is_pre_election: true
I20260929 18:58:47.926419 7091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d724c2210e034e1795ab197c8ad0ed0d in term 2.
I20260929 18:58:47.926578 7253 leader_election.cc:302] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f603fad9fd24ac282b9c3df456c8905, d724c2210e034e1795ab197c8ad0ed0d; no voters:
I20260929 18:58:47.926738 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 18:58:47.926822 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:47.926855 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:58:47.926896 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.926779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58424)
0929 18:58:47.926818 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.926851 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 18:58:47.927049 7253 leader_election.cc:334] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:47.927820 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:47.927966 7803 leader_election.cc:288] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 election: Requested vote from peers 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813), 2f603fad9fd24ac282b9c3df456c8905 (127.0.98.61:39317)
I20260929 18:58:47.928192 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.928066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58428)
0929 18:58:47.928108 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.928158 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 18:58:47.928275 7091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d724c2210e034e1795ab197c8ad0ed0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f603fad9fd24ac282b9c3df456c8905"
I20260929 18:58:47.928376 7091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:58:47.928525 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:47.928453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58438)
0929 18:58:47.928482 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:47.928505 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 18:58:47.928651 7253 leader_election.cc:334] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
I20260929 18:58:47.929150 7091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d724c2210e034e1795ab197c8ad0ed0d in term 3.
I20260929 18:58:47.929304 7253 leader_election.cc:302] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f603fad9fd24ac282b9c3df456c8905, d724c2210e034e1795ab197c8ad0ed0d; no voters: 5a6b66361c5b4b6c9f6bfbd1db7d15db
I20260929 18:58:47.929378 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:58:47.929481 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [term 3 LEADER]: Becoming Leader. State: Replica: d724c2210e034e1795ab197c8ad0ed0d, State: Running, Role: LEADER
I20260929 18:58:47.929553 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } }
I20260929 18:58:47.929913 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d724c2210e034e1795ab197c8ad0ed0d. Latest consensus state: current_term: 3 leader_uuid: "d724c2210e034e1795ab197c8ad0ed0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:47.930022 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:47.930195 7808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:47.930598 7808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:47.930688 7808 catalog_manager.cc:1259] Loaded cluster ID: 0a20dd9ee4fd4d9a91ab02a7950853cb
I20260929 18:58:47.930721 7808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:47.930927 7808 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:47.931026 7808 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Loaded TSK: 0
I20260929 18:58:47.931224 7808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:58:47.999730 7268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:48.001482 7268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:48.002772 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.002632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58452)
0929 18:58:48.002702 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.002744 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:58:48.002774 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [term 3 FOLLOWER]: Refusing update from remote peer d724c2210e034e1795ab197c8ad0ed0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 18:58:48.003222 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.003058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.0.1:58456)
0929 18:58:48.003105 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.003203 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:58:48.003158 7806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 18:58:48.003343 7253 consensus_peers.cc:487] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d -> Peer 5a6b66361c5b4b6c9f6bfbd1db7d15db (127.0.98.62:39813): Couldn't send request to peer 5a6b66361c5b4b6c9f6bfbd1db7d15db. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:58:48.003916 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d724c2210e034e1795ab197c8ad0ed0d. Latest consensus state: current_term: 3 leader_uuid: "d724c2210e034e1795ab197c8ad0ed0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:48.004038 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:58:48.004387 7809 mvcc.cc:205] Tried to move back new op lower bound from 7334741311496486912 to 7334741311200137216. Current Snapshot: MvccSnapshot[applied={T|T < 7334741311496486912}]
I20260929 18:58:48.004520 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d724c2210e034e1795ab197c8ad0ed0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:48.004617 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:48.004642 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d724c2210e034e1795ab197c8ad0ed0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5a6b66361c5b4b6c9f6bfbd1db7d15db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } peers { permanent_uuid: "2f603fad9fd24ac282b9c3df456c8905" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "d724c2210e034e1795ab197c8ad0ed0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36863 } attrs { promote: false } } }
I20260929 18:58:48.004732 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f603fad9fd24ac282b9c3df456c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:58:48.010164 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.0.98.1:33879)
0929 18:58:48.006959 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:48.006962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:48.006971 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:48.006997 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:48.006998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:48.007012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:48.007256 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:48.007317 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.008097 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009282 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.009848 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009920 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.009932 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:48.010078 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:58:48.010110 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":54,"threads_started":1}
I20260929 18:58:48.010188 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33879 (local address 127.0.0.1:56170)
0929 18:58:48.006764 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.006775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:48.006828 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:48.007117 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:48.007118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:48.007121 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:48.007186 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.007188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.008170 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.008171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.009205 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.009207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.009931 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.009999 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.010003 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:48.010070 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:58:48.010116 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 18:58:48.010407 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 (DEFAULT_TABLE table=test-table [id=beb5c9746d074272bf7a5952e76ef56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:48.010524 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1036e3cf61174ef7ab7656bcf1333d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:48.010542 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34925 (local address 127.0.0.1:38706)
0929 18:58:48.006762 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.006775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:48.006828 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:48.007326 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:48.007327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:48.007328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:48.007369 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.007370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.007993 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.007994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.009165 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.009167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.010338 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.010339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.010396 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.010399 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:48.010442 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 18:58:48.010480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260929 18:58:48.010727 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.2:34925)
0929 18:58:48.007149 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:48.007151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:48.007159 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:48.007196 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:48.007197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:48.007208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:48.007302 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:48.007429 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.007908 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009267 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.009841 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009920 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.009932 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:48.010617 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 18:58:48.010659 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":65,"threads_started":1}
I20260929 18:58:48.011106 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.98.3:36747)
0929 18:58:48.007057 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:48.007060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:48.007067 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:48.007093 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:48.007094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:48.007104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:48.007170 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:48.007376 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.008090 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.010166 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.010766 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.010838 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.010850 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:48.010998 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:58:48.011041 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260929 18:58:48.011104 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.006651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36747 (local address 127.0.0.1:57286)
0929 18:58:48.006963 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.006973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:48.007014 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:48.007217 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:48.007218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:48.007220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:48.007291 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.007292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.009289 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.009290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.010082 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:48.010083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.010836 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.010837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:48.010904 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.010908 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:48.010990 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:58:48.011033 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":72,"threads_started":1}
I20260929 18:58:48.011219 7518 tablet_service.cc:1467] Processing CreateTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 (DEFAULT_TABLE table=test-table [id=beb5c9746d074272bf7a5952e76ef56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:48.011358 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1036e3cf61174ef7ab7656bcf1333d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:48.011343 7654 tablet_service.cc:1467] Processing CreateTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 (DEFAULT_TABLE table=test-table [id=beb5c9746d074272bf7a5952e76ef56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:48.011479 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1036e3cf61174ef7ab7656bcf1333d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:48.012660 7821 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Bootstrap starting.
I20260929 18:58:48.012920 7820 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a: Bootstrap starting.
I20260929 18:58:48.013139 7821 tablet_bootstrap.cc:654] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:48.013306 7820 tablet_bootstrap.cc:654] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:48.013427 7822 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Bootstrap starting.
I20260929 18:58:48.013693 7821 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: No bootstrap required, opened a new log
I20260929 18:58:48.013746 7821 ts_tablet_manager.cc:1397] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:48.013857 7820 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a: No bootstrap required, opened a new log
I20260929 18:58:48.013899 7820 ts_tablet_manager.cc:1397] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:48.013883 7821 raft_consensus.cc:348] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.013957 7821 raft_consensus.cc:374] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:48.013975 7821 raft_consensus.cc:729] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18822a622364ca796a5484927e6ae22, State: Initialized, Role: FOLLOWER
I20260929 18:58:48.014027 7821 consensus_queue.cc:260] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.014003 7820 raft_consensus.cc:348] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.014082 7820 raft_consensus.cc:374] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:48.014108 7820 raft_consensus.cc:729] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4285b7f578a0453d9420b21e9346972a, State: Initialized, Role: FOLLOWER
I20260929 18:58:48.014159 7820 consensus_queue.cc:260] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.014211 7821 ts_tablet_manager.cc:1428] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:48.014312 7820 ts_tablet_manager.cc:1428] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:48.014544 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.014435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:34259)
0929 18:58:48.014481 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.014519 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 18:58:48.014761 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.014671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:36875)
0929 18:58:48.014708 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.014735 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 18:58:48.015246 7447 heartbeater.cc:503] Master 127.0.98.60:36863 was elected leader, sending a full tablet report...
I20260929 18:58:48.015300 7584 heartbeater.cc:503] Master 127.0.98.60:36863 was elected leader, sending a full tablet report...
I20260929 18:58:48.015322 7822 tablet_bootstrap.cc:654] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:48.015378 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.015246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:57797)
0929 18:58:48.015326 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.015352 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 18:58:48.015954 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.015847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:43563)
0929 18:58:48.015906 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.015932 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 18:58:48.016331 7822 tablet_bootstrap.cc:492] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: No bootstrap required, opened a new log
I20260929 18:58:48.016377 7822 ts_tablet_manager.cc:1397] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 18:58:48.016455 7822 raft_consensus.cc:348] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.016543 7822 raft_consensus.cc:374] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:48.016558 7822 raft_consensus.cc:729] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb93a0c7444beb9f83f0180302db3c, State: Initialized, Role: FOLLOWER
I20260929 18:58:48.016637 7822 consensus_queue.cc:260] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.016798 7822 ts_tablet_manager.cc:1428] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:48.017153 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.017048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:33355)
0929 18:58:48.017078 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.017105 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 18:58:48.017385 7720 heartbeater.cc:503] Master 127.0.98.60:36863 was elected leader, sending a full tablet report...
I20260929 18:58:48.045131 7828 raft_consensus.cc:482] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:48.045305 7828 raft_consensus.cc:504] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.045529 7828 leader_election.cc:288] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879), a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:48.045749 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1036e3cf61174ef7ab7656bcf1333d49" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4285b7f578a0453d9420b21e9346972a" is_pre_election: true
I20260929 18:58:48.045866 7401 raft_consensus.cc:2454] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 0.
I20260929 18:58:48.045894 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1036e3cf61174ef7ab7656bcf1333d49" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18822a622364ca796a5484927e6ae22" is_pre_election: true
I20260929 18:58:48.046005 7538 raft_consensus.cc:2454] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 0.
I20260929 18:58:48.046016 7609 leader_election.cc:302] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4285b7f578a0453d9420b21e9346972a, f9fb93a0c7444beb9f83f0180302db3c; no voters:
I20260929 18:58:48.046118 7828 raft_consensus.cc:2781] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:48.046181 7828 raft_consensus.cc:482] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:48.046208 7828 raft_consensus.cc:3037] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:48.046888 7828 raft_consensus.cc:504] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.047019 7828 leader_election.cc:288] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 election: Requested vote from peers 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879), a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925)
I20260929 18:58:48.047129 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1036e3cf61174ef7ab7656bcf1333d49" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4285b7f578a0453d9420b21e9346972a"
I20260929 18:58:48.047181 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1036e3cf61174ef7ab7656bcf1333d49" candidate_uuid: "f9fb93a0c7444beb9f83f0180302db3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18822a622364ca796a5484927e6ae22"
I20260929 18:58:48.047261 7401 raft_consensus.cc:3037] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:48.047260 7538 raft_consensus.cc:3037] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:48.047874 7401 raft_consensus.cc:2454] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 1.
I20260929 18:58:48.047892 7538 raft_consensus.cc:2454] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb93a0c7444beb9f83f0180302db3c in term 1.
I20260929 18:58:48.048092 7607 leader_election.cc:302] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a18822a622364ca796a5484927e6ae22, f9fb93a0c7444beb9f83f0180302db3c; no voters:
I20260929 18:58:48.048216 7828 raft_consensus.cc:2781] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:48.048401 7828 raft_consensus.cc:686] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 1 LEADER]: Becoming Leader. State: Replica: f9fb93a0c7444beb9f83f0180302db3c, State: Running, Role: LEADER
I20260929 18:58:48.048449 7828 consensus_queue.cc:237] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260929 18:58:48.048820 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.048649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:38663)
0929 18:58:48.048744 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.048782 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 18:58:48.049235 7267 catalog_manager.cc:5495] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c reported cstate change: term changed from 0 to 1, leader changed from <none> to f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f9fb93a0c7444beb9f83f0180302db3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9fb93a0c7444beb9f83f0180302db3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } health_report { overall_health: HEALTHY } } }
I20260929 18:58:48.057163 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.054819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.98.3:36747)
0929 18:58:48.054878 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:48.054880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:48.054888 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:48.054957 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:48.054958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:48.054970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:48.055026 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:48.055130 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.055671 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.056442 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:48.056652 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:48.056741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:48.056897 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:48.056930 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:48.057000 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:48.057061 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:58:48.057082 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:58:48.058002 7401 raft_consensus.cc:1261] T 1036e3cf61174ef7ab7656bcf1333d49 P 4285b7f578a0453d9420b21e9346972a [term 1 FOLLOWER]: Refusing update from remote peer f9fb93a0c7444beb9f83f0180302db3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:48.058202 7538 raft_consensus.cc:1261] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Refusing update from remote peer f9fb93a0c7444beb9f83f0180302db3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:48.058295 7828 consensus_queue.cc:1035] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4285b7f578a0453d9420b21e9346972a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:48.058437 7829 consensus_queue.cc:1035] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18822a622364ca796a5484927e6ae22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:48.059249 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.059130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:42609)
0929 18:58:48.059186 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.059217 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:58:48.059459 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.059363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:45897)
0929 18:58:48.059401 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.059430 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 18:58:48.059715 7833 mvcc.cc:205] Tried to move back new op lower bound from 7334741311723925504 to 7334741311687090176. Current Snapshot: MvccSnapshot[applied={T|T < 7334741311723925504}]
I20260929 18:58:48.060086 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.059922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:58153)
0929 18:58:48.060025 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.060052 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 18:58:48.060192 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.060113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.1:41039)
0929 18:58:48.060148 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.060170 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 18:58:48.061280 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.061076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.3:52515)
0929 18:58:48.061224 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.061253 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 18:58:48.061980 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.061866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39813 (local address 127.0.98.2:40815)
0929 18:58:48.061922 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.061950 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:58:48.062036 7268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:48.061985 7835 mvcc.cc:205] Tried to move back new op lower bound from 7334741311723925504 to 7334741311687090176. Current Snapshot: MvccSnapshot[applied={T|T < 7334741311723925504 or (T in {7334741311723925504})}]
I20260929 18:58:48.062129 7268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:48.063436 7268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d724c2210e034e1795ab197c8ad0ed0d: Sending DeleteTablet for 3 replicas of tablet 1036e3cf61174ef7ab7656bcf1333d49
I20260929 18:58:48.063728 7518 tablet_service.cc:1514] Processing DeleteTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38706
I20260929 18:58:48.063802 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57286
I20260929 18:58:48.063795 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 1036e3cf61174ef7ab7656bcf1333d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56170
I20260929 18:58:48.063946 7842 tablet_replica.cc:330] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: stopping tablet replica
I20260929 18:58:48.064006 7842 raft_consensus.cc:2229] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:48.064021 7844 tablet_replica.cc:330] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: stopping tablet replica
I20260929 18:58:48.064055 7842 raft_consensus.cc:2258] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:48.064093 7844 raft_consensus.cc:2229] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:48.064165 7844 raft_consensus.cc:2258] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:48.064199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7229
I20260929 18:58:48.064919 7844 ts_tablet_manager.cc:1905] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:48.065191 7842 ts_tablet_manager.cc:1905] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:48.066849 7842 ts_tablet_manager.cc:1918] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:48.066843 7844 ts_tablet_manager.cc:1918] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:48.066929 7844 log.cc:1196] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/1036e3cf61174ef7ab7656bcf1333d49
I20260929 18:58:48.066931 7842 log.cc:1196] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/1036e3cf61174ef7ab7656bcf1333d49
I20260929 18:58:48.067149 7842 ts_tablet_manager.cc:1939] T 1036e3cf61174ef7ab7656bcf1333d49 P a18822a622364ca796a5484927e6ae22: Deleting consensus metadata
I20260929 18:58:48.067233 7844 ts_tablet_manager.cc:1939] T 1036e3cf61174ef7ab7656bcf1333d49 P f9fb93a0c7444beb9f83f0180302db3c: Deleting consensus metadata
I20260929 18:58:48.067500 7252 catalog_manager.cc:4841] TS a18822a622364ca796a5484927e6ae22 (127.0.98.2:34925): tablet 1036e3cf61174ef7ab7656bcf1333d49 (table test-table [id=beb5c9746d074272bf7a5952e76ef56a]) successfully deleted
I20260929 18:58:48.067543 7254 catalog_manager.cc:4841] TS f9fb93a0c7444beb9f83f0180302db3c (127.0.98.3:36747): tablet 1036e3cf61174ef7ab7656bcf1333d49 (table test-table [id=beb5c9746d074272bf7a5952e76ef56a]) successfully deleted
I20260929 18:58:48.071115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7455
I20260929 18:58:48.071153 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:48.070964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33879 (local address 127.0.0.1:56178)
0929 18:58:48.071073 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:48.071110 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 18:58:48.072091 7254 catalog_manager.cc:4584] TS 4285b7f578a0453d9420b21e9346972a (127.0.98.1:33879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1036e3cf61174ef7ab7656bcf1333d49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33879: connect: Connection refused (error 111)
I20260929 18:58:48.076946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7591
I20260929 18:58:48.082875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 6947
I20260929 18:58:48.087929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7135
2026-09-29T18:58:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6148 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:48 dist-test-slave-gfsd krb5kdc[7847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:48 dist-test-slave-gfsd krb5kdc[7847](info): set up 2 sockets
Sep 29 18:58:48 dist-test-slave-gfsd krb5kdc[7847](info): commencing operation
krb5kdc: starting...
W20260929 18:58:49.119802 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 18:58:49.238503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45025
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:49.319172 7860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:49.319433 7860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:49.319473 7860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:49.320628 7860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:49.320698 7860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:58:49.320721 7860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:49.320753 7860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:49.320778 7860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:49.320808 7860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:49.322242 7860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:49.322499 7860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:49.322719 7860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:49.327756 7866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.327791 7860 server_base.cc:1034] running on GCE node
W20260929 18:58:49.327764 7865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:49.327773 7868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.328235 7860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:49.328501 7860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:49.329620 7860 hybrid_clock.cc:648] HybridClock initialized: now 1790708329329596 us; error 49 us; skew 500 ppm
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:49.332056 7860 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 18:58:49.332485 7860 webserver.cc:460] Webserver started at http://127.0.98.62:33941/ using document root <none> and password file <none>
I20260929 18:58:49.332762 7860 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:49.332813 7860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:49.332959 7860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:49.333992 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "2656c285ea82433a9fac550f47496a29"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "846de6d4ca0e3f06e1a48595a8e0e2e0"
server_key_iv: "8fd2a18c28eee48d6602f07777135098"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.334342 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "2656c285ea82433a9fac550f47496a29"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "846de6d4ca0e3f06e1a48595a8e0e2e0"
server_key_iv: "8fd2a18c28eee48d6602f07777135098"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.335692 7860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:58:49.336438 7875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.336644 7860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:58:49.336750 7860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "2656c285ea82433a9fac550f47496a29"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "846de6d4ca0e3f06e1a48595a8e0e2e0"
server_key_iv: "8fd2a18c28eee48d6602f07777135098"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.336843 7860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:49.349503 7860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:49.351547 7860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:49.351701 7860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:49.356256 7860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45025
I20260929 18:58:49.356309 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45025 every 8 connection(s)
I20260929 18:58:49.356624 7860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:49.357290 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:49.359504 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Bootstrap starting.
I20260929 18:58:49.360074 7938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:49.360375 7938 log.cc:824] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:49.361009 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: No bootstrap required, opened a new log
I20260929 18:58:49.362954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7860
I20260929 18:58:49.363067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:49.363265 392 external_mini_cluster.cc:1428] Setting key ae47ccfee024152ccb8eafbf82cac8ca
I20260929 18:58:49.363322 7938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260929 18:58:49.363497 7938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:49.363554 7938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2656c285ea82433a9fac550f47496a29, State: Initialized, Role: FOLLOWER
I20260929 18:58:49.363724 7938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260929 18:58:49.363822 7938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:49.363880 7938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:49.363972 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:49.364634 7938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260929 18:58:49.364717 7938 leader_election.cc:302] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2656c285ea82433a9fac550f47496a29; no voters:
I20260929 18:58:49.364880 7938 leader_election.cc:288] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:49.364903 7943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:49.365000 7943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [term 1 LEADER]: Becoming Leader. State: Replica: 2656c285ea82433a9fac550f47496a29, State: Running, Role: LEADER
I20260929 18:58:49.365077 7943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260929 18:58:49.365136 7938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:49.365396 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2656c285ea82433a9fac550f47496a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } } }
I20260929 18:58:49.365545 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:49.365746 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2656c285ea82433a9fac550f47496a29. Latest consensus state: current_term: 1 leader_uuid: "2656c285ea82433a9fac550f47496a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2656c285ea82433a9fac550f47496a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } } }
I20260929 18:58:49.365818 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:49.366329 7951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:49.366873 7951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:49.368570 7951 catalog_manager.cc:1347] Generated new cluster ID: c5bcefdbe3234176acf7aadce165b9cb
I20260929 18:58:49.368624 7951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:49.374269 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.363781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.98.62:45025)
0929 18:58:49.363980 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.363990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.364024 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:58:49.364145 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.364151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.364205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.364347 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.365039 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.366500 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.367591 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.368220 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.368333 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.370640 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.370644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.370644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.370655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.372799 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.373182 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.373185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.373186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.373284 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.373491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.373492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.373492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.373728 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.373908 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.373972 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.374107 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260929 18:58:49.381515 7951 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:49.382004 7951 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:49.386279 7951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Generated new TSK 0
I20260929 18:58:49.386449 7951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 18:58:49.415637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:49.518424 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:49.518630 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:49.518654 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:49.519870 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:49.519913 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:49.519934 7964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:49.519951 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:49.521447 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:49.521682 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:49.521920 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:49.527706 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:49.527801 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:49.527707 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.527937 7964 server_base.cc:1034] running on GCE node
I20260929 18:58:49.528155 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:49.528398 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:49.529558 7964 hybrid_clock.cc:648] HybridClock initialized: now 1790708329529542 us; error 32 us; skew 500 ppm
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:49.532172 7964 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:58:49.532712 7964 webserver.cc:460] Webserver started at http://127.0.98.1:41401/ using document root <none> and password file <none>
I20260929 18:58:49.532990 7964 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:49.533057 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:49.533211 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:49.534271 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2ebfad725ef479cb9d833f2bf96a525"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "c238c103091197eb102ca77dbf8a1f3a"
server_key_iv: "29d09e6542b4e1c9d9917f3ad0c72e08"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.534638 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2ebfad725ef479cb9d833f2bf96a525"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "c238c103091197eb102ca77dbf8a1f3a"
server_key_iv: "29d09e6542b4e1c9d9917f3ad0c72e08"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.536011 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:49.536755 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.536989 7964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:49.537096 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "e2ebfad725ef479cb9d833f2bf96a525"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "c238c103091197eb102ca77dbf8a1f3a"
server_key_iv: "29d09e6542b4e1c9d9917f3ad0c72e08"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.537156 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:49.551231 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:49.553524 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:49.553699 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:49.554184 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:49.554250 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.554294 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:49.554320 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.561035 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41411
I20260929 18:58:49.561092 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41411 every 8 connection(s)
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:49.561430 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:49.564639 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.554779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43873 (local address 127.0.98.62:45025)
0929 18:58:49.554865 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.554867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.555055 (+ 188us) server_negotiation.cc:407] Connection header received
0929 18:58:49.555626 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.555627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.555644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.555725 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.556558 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.557435 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.559013 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.559381 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.559468 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.561947 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.561949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.561950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.561961 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.563175 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.563613 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.563614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.563615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.563690 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.563942 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.563943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.563944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.564065 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.564151 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.564427 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.564539 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:58:49.564788 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.554740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.1:43873)
0929 18:58:49.554920 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.554961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.555522 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.555810 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.555814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.556171 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.556440 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.556446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.557534 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.557535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.558904 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.558907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.559475 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.559476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.559575 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.560124 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.560135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.561863 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.563261 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.563268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.563270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.563530 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.563758 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.563759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.563760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.563873 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.564143 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.564149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.564288 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.564394 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.564549 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260929 18:58:49.565275 8095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260929 18:58:49.565363 8095 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:49.565544 8095 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260929 18:58:49.566207 7892 ts_manager.cc:194] Registered new tserver with Master: e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411)
I20260929 18:58:49.566896 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43873
I20260929 18:58:49.571110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 7964
I20260929 18:58:49.571252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:49.571496 392 external_mini_cluster.cc:1428] Setting key e812eb29233bbdc13a068d5795a03510
I20260929 18:58:49.572094 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.566267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.1:36511)
0929 18:58:49.566452 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.566461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.566535 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.566665 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.566665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.566825 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.566904 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.566905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.567676 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.567677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.568179 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.568180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.568562 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.568563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.568634 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.569001 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.569005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.569343 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.570935 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.570936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.570936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.571116 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.571351 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.571352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.571352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.571410 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.571754 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.571755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.571820 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.571898 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.571974 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260929 18:58:49.572300 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.566328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36511 (local address 127.0.98.62:45025)
0929 18:58:49.566379 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.566381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.566522 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:58:49.566592 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.566593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.566603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.566659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.566963 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.567572 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.568275 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.568480 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.568554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.569473 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.569474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.569475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.569481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.570819 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.571211 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.571212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.571213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.571264 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.571492 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.571493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.571493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.571659 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.571735 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.571942 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.572019 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 18:58:49.598776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:49.680882 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:49.681074 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:49.681097 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:49.682157 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:49.682189 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:49.682209 8104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:49.682225 8104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:49.683578 8104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:49.683745 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:49.683909 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:49.688782 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.688853 8104 server_base.cc:1034] running on GCE node
W20260929 18:58:49.688805 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:49.688831 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.689188 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:49.689429 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:49.690596 8104 hybrid_clock.cc:648] HybridClock initialized: now 1790708329690578 us; error 50 us; skew 500 ppm
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:49.693150 8104 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:58:49.693562 8104 webserver.cc:460] Webserver started at http://127.0.98.2:36213/ using document root <none> and password file <none>
I20260929 18:58:49.693802 8104 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:49.693871 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:49.694023 8104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:49.695050 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "886a0f71db9f4ebaa254b78630b27a4e"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "612abe0e2373dbfbb5d525443033aaf1"
server_key_iv: "537c4290629dd13b0889864b037f1ff0"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.695423 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "886a0f71db9f4ebaa254b78630b27a4e"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "612abe0e2373dbfbb5d525443033aaf1"
server_key_iv: "537c4290629dd13b0889864b037f1ff0"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.696803 8104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:49.697525 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.697703 8104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:58:49.697846 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "886a0f71db9f4ebaa254b78630b27a4e"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "612abe0e2373dbfbb5d525443033aaf1"
server_key_iv: "537c4290629dd13b0889864b037f1ff0"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.697954 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:49.733459 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:49.735848 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:49.736009 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:49.736524 8104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:49.736588 8104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.736656 8104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:49.736686 8104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:49.744107 8104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43519
I20260929 18:58:49.744180 8234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43519 every 8 connection(s)
I20260929 18:58:49.744490 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:49.746712 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.737083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38509 (local address 127.0.98.62:45025)
0929 18:58:49.737154 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.737156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.737416 (+ 260us) server_negotiation.cc:407] Connection header received
0929 18:58:49.737953 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.737954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.737965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.738035 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.739001 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.739806 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.741020 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.741235 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.741330 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.743793 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.743795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.743795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.743802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.745354 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.745769 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.745770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.745771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.745829 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.746082 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.746083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.746084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.746196 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.746270 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.746538 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.746622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:58:49.746904 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.737069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.2:38509)
0929 18:58:49.737303 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.737342 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.737858 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.738174 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.738179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.738570 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.738879 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.738887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.739911 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.739913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.740922 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.740925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.741352 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.741353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.741486 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.742052 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.742064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.743682 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.745452 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.745457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.745460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.745679 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.745896 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.745897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.745897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.746005 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.746262 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.746267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.746405 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.746515 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.746667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":174,"threads_started":1}
I20260929 18:58:49.747404 8235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260929 18:58:49.747507 8235 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:49.747675 8235 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260929 18:58:49.748132 7891 ts_manager.cc:194] Registered new tserver with Master: 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519)
I20260929 18:58:49.748680 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38509
I20260929 18:58:49.753661 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.748435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51725 (local address 127.0.98.62:45025)
0929 18:58:49.748468 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.748470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.748579 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:58:49.748610 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.748611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.748619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.748684 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.749026 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.749482 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.750197 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.750402 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.750484 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.751335 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.751336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.751336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.751343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.752500 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.752851 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.752852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.752852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.752908 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.753108 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.753109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.753109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.753250 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.753328 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.753474 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.753546 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:58:49.753650 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.748374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.2:51725)
0929 18:58:49.748496 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.748506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.748570 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.748687 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.748687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.748891 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.748949 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.748951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.749575 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.749576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.750106 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.750108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.750487 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.750488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.750554 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.750901 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.750905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.751222 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.752596 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.752597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.752598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.752766 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.752977 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.752978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.752978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.753035 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.753319 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.753320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.753361 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.753450 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.753546 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260929 18:58:49.753932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8104
I20260929 18:58:49.754053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:49.754273 392 external_mini_cluster.cc:1428] Setting key 4b0094240959f1d19fff0f6e1a1980db
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:58:49.778616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:49.864869 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:49.865090 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:49.865114 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:49.866693 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:49.866761 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:49.866799 8244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:58:49.866822 8244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:49.868340 8244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:49.868525 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:49.868696 8244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:49.873893 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:49.873893 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.874315 8244 server_base.cc:1034] running on GCE node
W20260929 18:58:49.873919 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:49.874588 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:49.874805 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:49.875948 8244 hybrid_clock.cc:648] HybridClock initialized: now 1790708329875939 us; error 27 us; skew 500 ppm
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:49.878578 8244 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:58:49.879032 8244 webserver.cc:460] Webserver started at http://127.0.98.3:39061/ using document root <none> and password file <none>
I20260929 18:58:49.879217 8244 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:49.879257 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:49.879418 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:49.880411 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "164aef27824ffb2759438187dc8a40de"
server_key_iv: "a26e5af0fa77aa5972b9a752552e0ea5"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.880703 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "164aef27824ffb2759438187dc8a40de"
server_key_iv: "a26e5af0fa77aa5972b9a752552e0ea5"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.881927 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:58:49.882634 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.882828 8244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:49.882925 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
format_stamp: "Formatted at 2026-09-29 18:58:49 on dist-test-slave-gfsd"
server_key: "164aef27824ffb2759438187dc8a40de"
server_key_iv: "a26e5af0fa77aa5972b9a752552e0ea5"
server_key_version: "encryptionkey@0"
I20260929 18:58:49.883008 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:49.908649 8244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:49.910671 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:49.910835 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:49.911460 8244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:49.911512 8244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:49.911541 8244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:49.911562 8244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:49.918751 8244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42997
I20260929 18:58:49.918789 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42997 every 8 connection(s)
I20260929 18:58:49.919162 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:49.921094 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.911878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32937 (local address 127.0.98.62:45025)
0929 18:58:49.911949 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.911952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.912153 (+ 201us) server_negotiation.cc:407] Connection header received
0929 18:58:49.912641 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.912643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.912654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.912746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.913449 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.914129 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.915516 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.915723 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.915801 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.918289 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.918291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.918292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.918301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.919951 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.920332 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.920332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.920333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.920383 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.920587 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.920587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.920588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.920696 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.920745 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.920974 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.921033 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:58:49.921296 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.911852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.3:32937)
0929 18:58:49.912019 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.912060 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.912547 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.912810 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.912813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.913081 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.913349 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.913353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.914258 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.914260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.915411 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.915419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.915807 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.915808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.915938 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.916424 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.916433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.918186 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.920022 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.920027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.920028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.920264 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.920439 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.920440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.920440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.920533 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.920751 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.920755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.920864 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.920945 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.921057 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260929 18:58:49.921754 8375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260929 18:58:49.921860 8375 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:49.922026 8375 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260929 18:58:49.922363 7892 ts_manager.cc:194] Registered new tserver with Master: 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:49.922806 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32937
I20260929 18:58:49.923355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8244
I20260929 18:58:49.923462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:49.923671 392 external_mini_cluster.cc:1428] Setting key 3c60c50da865d10d7369abadf6a06af4
I20260929 18:58:49.924599 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:58:49.929652 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.922768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33403 (local address 127.0.98.62:45025)
0929 18:58:49.922810 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.922812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.922920 (+ 108us) server_negotiation.cc:407] Connection header received
0929 18:58:49.922997 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.922997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.923008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.923077 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.924147 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.924846 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.925589 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.925889 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.925962 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.927137 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.927138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.927138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.927145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.928519 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.928889 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.928890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.928891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.928943 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.929150 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.929150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.929151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.929261 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.929335 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.929492 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.929563 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:58:49.929665 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.922711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.3:33403)
0929 18:58:49.922863 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.922873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.922929 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.923102 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.923103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.923998 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:49.924071 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.924073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.924960 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.924961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.925503 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.925505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.925986 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.925988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.926047 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.926650 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:49.926655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:49.926996 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:49.928629 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.928630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.928631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.928803 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.929019 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:49.929020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:49.929020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:49.929077 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:49.929328 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:49.929329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:49.929371 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.929468 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.929564 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:58:49.946498 7942 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:49.940977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.0.98.62:45025)
0929 18:58:49.941040 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.941042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.941067 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:58:49.941141 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.941142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.941151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.941197 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.941775 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.942251 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.943030 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.943222 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.943330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.946399 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37524: BlockingRecv error: recv got EOF from 127.0.0.1:37524 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:58:49 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:49.953811 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.947368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.0.98.62:45025)
0929 18:58:49.947412 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.947413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.947423 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:49.947465 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.947465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.947473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.947519 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:49.947952 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.948451 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.949144 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.949342 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.949417 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.951585 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:49.951587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:49.951587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:49.951593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:49.952878 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.953211 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.953212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.953212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.953263 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:49.953427 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:49.953427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:49.953428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:49.953527 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:49.953598 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.953684 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.953739 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:49.955417 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:49.957094 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:49.963726 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42997 (local address 127.0.0.1:57638)
0929 18:58:49.959678 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.959725 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.959789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.960139 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.960142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.960144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:49.960260 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.960262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961594 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.961595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.962499 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.962501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963442 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.963443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.963521 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.963525 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.963555 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.963621 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":7,"thread_start_us":50,"threads_started":1}
I20260929 18:58:49.963759 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.0.98.2:43519)
0929 18:58:49.959781 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.959788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.959908 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:58:49.960054 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.960058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.960079 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.960217 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:49.960363 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961490 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.962777 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963251 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.963377 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.963462 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.963611 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.963651 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260929 18:58:49.963737 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43519 (local address 127.0.0.1:47618)
0929 18:58:49.959839 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.959852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.959937 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.960211 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.960212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.960214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:49.960281 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.960283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961574 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.961574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.962677 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.962679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963385 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.963386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.963480 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.963494 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.963595 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.963637 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 18:58:49.964016 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.98.1:41411)
0929 18:58:49.960149 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.960153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.960162 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:49.960196 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.960198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.960214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.960304 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:49.960486 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961222 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.962721 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963466 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.963524 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.963605 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.963842 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.963896 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260929 18:58:49.964385 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.98.3:42997)
0929 18:58:49.959960 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:49.959965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:49.959974 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:49.960011 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:49.960014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:49.960034 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:49.960116 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:49.960328 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961477 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.962576 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963337 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.964188 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.964266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:49.964290 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 18:58:49.964320 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":96,"threads_started":1}
I20260929 18:58:49.964524 8169 tablet_service.cc:1467] Processing CreateTablet for tablet e11918b9af86477a9b4737ab471371db (DEFAULT_TABLE table=test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:49.964867 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:49.959727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41411 (local address 127.0.0.1:36764)
0929 18:58:49.959867 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:49.959877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:49.959922 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:49.960366 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:49.960366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:49.960368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:49.960411 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.960413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.961322 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.961323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.962630 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:49.962633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:49.963637 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:49.963638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:49.963701 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:49.963704 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:49.963753 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:58:49.963793 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260929 18:58:49.964931 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11918b9af86477a9b4737ab471371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:49.965051 8309 tablet_service.cc:1467] Processing CreateTablet for tablet e11918b9af86477a9b4737ab471371db (DEFAULT_TABLE table=test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:49.965386 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11918b9af86477a9b4737ab471371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:49.965775 8029 tablet_service.cc:1467] Processing CreateTablet for tablet e11918b9af86477a9b4737ab471371db (DEFAULT_TABLE table=test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:49.966142 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11918b9af86477a9b4737ab471371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:49.969192 8403 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Bootstrap starting.
I20260929 18:58:49.969275 8405 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Bootstrap starting.
I20260929 18:58:49.969353 8404 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Bootstrap starting.
I20260929 18:58:49.969843 8405 tablet_bootstrap.cc:654] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:49.969908 8404 tablet_bootstrap.cc:654] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:49.970139 8405 log.cc:824] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:49.970207 8404 log.cc:824] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:49.970960 8404 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: No bootstrap required, opened a new log
I20260929 18:58:49.970980 8403 tablet_bootstrap.cc:654] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:49.970981 8405 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: No bootstrap required, opened a new log
I20260929 18:58:49.971098 8404 ts_tablet_manager.cc:1397] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:49.971117 8405 ts_tablet_manager.cc:1397] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:49.971379 8403 log.cc:824] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:49.972133 8403 tablet_bootstrap.cc:492] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: No bootstrap required, opened a new log
I20260929 18:58:49.972200 8403 ts_tablet_manager.cc:1397] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 18:58:49.972810 8404 raft_consensus.cc:348] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.972963 8404 raft_consensus.cc:374] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:49.973006 8404 raft_consensus.cc:729] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Initialized, Role: FOLLOWER
I20260929 18:58:49.973116 8404 consensus_queue.cc:260] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.973213 8405 raft_consensus.cc:348] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.973373 8405 raft_consensus.cc:374] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:49.973397 8404 ts_tablet_manager.cc:1428] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:58:49.973414 8405 raft_consensus.cc:729] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Initialized, Role: FOLLOWER
I20260929 18:58:49.973517 8375 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
I20260929 18:58:49.973548 8405 consensus_queue.cc:260] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.973819 8405 ts_tablet_manager.cc:1428] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:58:49.973862 8095 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
I20260929 18:58:49.974197 8403 raft_consensus.cc:348] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.974336 8403 raft_consensus.cc:374] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:49.974376 8403 raft_consensus.cc:729] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Initialized, Role: FOLLOWER
I20260929 18:58:49.974491 8403 consensus_queue.cc:260] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:49.974761 8403 ts_tablet_manager.cc:1428] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:58:49.974934 8235 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
W20260929 18:58:49.995024 8236 tablet.cc:2369] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:49.999864 8410 raft_consensus.cc:482] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:50.000036 8410 raft_consensus.cc:504] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:50.000423 8410 leader_election.cc:288] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997), 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519)
I20260929 18:58:50.004225 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.000628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42997 (local address 127.0.98.1:46921)
0929 18:58:50.000691 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.000706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.000755 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.000954 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.000955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.000957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.001038 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.001040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.001980 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.001981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.003137 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.003139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.003953 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.003954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.004043 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.004048 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.004110 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.004153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 18:58:50.004245 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.000739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59227 (local address 127.0.98.2:43519)
0929 18:58:50.000785 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.000787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.001099 (+ 312us) server_negotiation.cc:407] Connection header received
0929 18:58:50.001215 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.001216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.001229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.001289 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.001468 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.002129 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.003364 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.003901 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.003988 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.003996 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.004139 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.004178 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:58:50.004245 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.000672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46921 (local address 127.0.98.3:42997)
0929 18:58:50.000728 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.000731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.000742 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:58:50.000841 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.000842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.000852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.000931 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.001146 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.001896 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.003236 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.003837 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.003924 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.003936 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.004140 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.004178 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:58:50.004343 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.000629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43519 (local address 127.0.98.1:59227)
0929 18:58:50.001038 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.001050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.001123 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.001297 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.001298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.001299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.001388 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.001389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.002207 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.002208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.003307 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.003309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.004007 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.004008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.004067 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.004070 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.004121 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.004214 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":203,"thread_start_us":115,"threads_started":1}
I20260929 18:58:50.004681 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11918b9af86477a9b4737ab471371db" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e" is_pre_election: true
I20260929 18:58:50.004842 8189 raft_consensus.cc:2454] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 0.
I20260929 18:58:50.004822 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11918b9af86477a9b4737ab471371db" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" is_pre_election: true
I20260929 18:58:50.004966 8329 raft_consensus.cc:2454] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 0.
I20260929 18:58:50.005084 7981 leader_election.cc:302] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886a0f71db9f4ebaa254b78630b27a4e, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.005218 8410 raft_consensus.cc:2781] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:50.005276 8410 raft_consensus.cc:482] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:50.005303 8410 raft_consensus.cc:3037] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.006248 8410 raft_consensus.cc:504] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:50.006397 8410 leader_election.cc:288] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 election: Requested vote from peers 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997), 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519)
I20260929 18:58:50.006577 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11918b9af86477a9b4737ab471371db" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e"
I20260929 18:58:50.006577 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11918b9af86477a9b4737ab471371db" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
I20260929 18:58:50.006650 8189 raft_consensus.cc:3037] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.006657 8329 raft_consensus.cc:3037] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.007421 8189 raft_consensus.cc:2454] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 1.
I20260929 18:58:50.007520 8329 raft_consensus.cc:2454] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 1.
I20260929 18:58:50.007632 7981 leader_election.cc:302] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886a0f71db9f4ebaa254b78630b27a4e, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.007721 8410 raft_consensus.cc:2781] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:50.007866 8410 raft_consensus.cc:686] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 1 LEADER]: Becoming Leader. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Running, Role: LEADER
I20260929 18:58:50.007978 8410 consensus_queue.cc:237] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } }
I20260929 18:58:50.008912 7892 catalog_manager.cc:5495] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2ebfad725ef479cb9d833f2bf96a525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } health_report { overall_health: HEALTHY } } }
I20260929 18:58:50.021155 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.018774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.98.1:41411)
0929 18:58:50.018836 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.018838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.018845 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:50.018873 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.018874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.018892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.018954 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.019065 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.019591 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.020402 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.020593 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.020650 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.020864 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.020958 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.021016 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.021068 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.021089 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:58:50.022927 8189 raft_consensus.cc:1261] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Refusing update from remote peer e2ebfad725ef479cb9d833f2bf96a525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.022959 8329 raft_consensus.cc:1261] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Refusing update from remote peer e2ebfad725ef479cb9d833f2bf96a525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.023335 8410 consensus_queue.cc:1035] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.023538 8413 consensus_queue.cc:1035] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.025493 8416 mvcc.cc:205] Tried to move back new op lower bound from 7334741319771058176 to 7334741319713218560. Current Snapshot: MvccSnapshot[applied={T|T < 7334741319771058176}]
I20260929 18:58:50.026077 8418 mvcc.cc:205] Tried to move back new op lower bound from 7334741319771058176 to 7334741319713218560. Current Snapshot: MvccSnapshot[applied={T|T < 7334741319771058176}]
I20260929 18:58:50.028245 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.028340 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.029295 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Sending DeleteTablet for 3 replicas of tablet e11918b9af86477a9b4737ab471371db
I20260929 18:58:50.029621 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet e11918b9af86477a9b4737ab471371db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57638
I20260929 18:58:50.029650 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet e11918b9af86477a9b4737ab471371db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 18:58:50.029696 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet e11918b9af86477a9b4737ab471371db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47618
I20260929 18:58:50.029827 8425 tablet_replica.cc:330] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: stopping tablet replica
I20260929 18:58:50.029887 8427 tablet_replica.cc:330] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: stopping tablet replica
I20260929 18:58:50.029910 8425 raft_consensus.cc:2229] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.029976 8427 raft_consensus.cc:2229] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.030102 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:50.030246 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:50.030229 8425 raft_consensus.cc:2258] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.030512 8427 raft_consensus.cc:2258] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.030786 8425 ts_tablet_manager.cc:1905] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.031014 8426 tablet_replica.cc:330] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: stopping tablet replica
I20260929 18:58:50.031071 8427 ts_tablet_manager.cc:1905] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.031188 8426 raft_consensus.cc:2229] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:50.031383 8426 raft_consensus.cc:2258] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.031821 8426 ts_tablet_manager.cc:1905] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.032318 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 1462daa5bccc498385e8ae802cb7a593 (DEFAULT_TABLE table=test-table [id=0325fd2a2d4947bbbdb1376bde732e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.032358 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 1462daa5bccc498385e8ae802cb7a593 (DEFAULT_TABLE table=test-table [id=0325fd2a2d4947bbbdb1376bde732e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.032449 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462daa5bccc498385e8ae802cb7a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.032524 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462daa5bccc498385e8ae802cb7a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.032905 8427 ts_tablet_manager.cc:1918] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.032963 8427 log.cc:1196] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/e11918b9af86477a9b4737ab471371db
I20260929 18:58:50.033212 8427 ts_tablet_manager.cc:1939] T e11918b9af86477a9b4737ab471371db P 886a0f71db9f4ebaa254b78630b27a4e: Deleting consensus metadata
I20260929 18:58:50.033632 7877 catalog_manager.cc:4841] TS 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519): tablet e11918b9af86477a9b4737ab471371db (table test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]) successfully deleted
I20260929 18:58:50.033865 8404 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Bootstrap starting.
I20260929 18:58:50.033960 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 1462daa5bccc498385e8ae802cb7a593 (DEFAULT_TABLE table=test-table [id=0325fd2a2d4947bbbdb1376bde732e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.034056 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462daa5bccc498385e8ae802cb7a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.034283 8404 tablet_bootstrap.cc:654] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.034567 8405 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Bootstrap starting.
I20260929 18:58:50.034964 8405 tablet_bootstrap.cc:654] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.035493 8425 ts_tablet_manager.cc:1918] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.035550 8405 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: No bootstrap required, opened a new log
I20260929 18:58:50.035563 8425 log.cc:1196] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/e11918b9af86477a9b4737ab471371db
I20260929 18:58:50.035591 8405 ts_tablet_manager.cc:1397] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:50.035725 8405 raft_consensus.cc:348] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.035792 8405 raft_consensus.cc:374] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.035815 8405 raft_consensus.cc:729] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.035817 8403 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Bootstrap starting.
I20260929 18:58:50.035871 8405 consensus_queue.cc:260] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.035965 8405 ts_tablet_manager.cc:1428] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.036120 8425 ts_tablet_manager.cc:1939] T e11918b9af86477a9b4737ab471371db P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting consensus metadata
I20260929 18:58:50.036204 8403 tablet_bootstrap.cc:654] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.036500 7878 catalog_manager.cc:4841] TS 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997): tablet e11918b9af86477a9b4737ab471371db (table test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]) successfully deleted
I20260929 18:58:50.036964 8403 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: No bootstrap required, opened a new log
I20260929 18:58:50.037021 8403 ts_tablet_manager.cc:1397] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:50.037171 8403 raft_consensus.cc:348] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.037226 8403 raft_consensus.cc:374] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.037240 8403 raft_consensus.cc:729] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.037302 8403 consensus_queue.cc:260] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.037405 8403 ts_tablet_manager.cc:1428] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.038772 8426 ts_tablet_manager.cc:1918] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.038828 8426 log.cc:1196] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/e11918b9af86477a9b4737ab471371db
I20260929 18:58:50.039116 8426 ts_tablet_manager.cc:1939] T e11918b9af86477a9b4737ab471371db P e2ebfad725ef479cb9d833f2bf96a525: Deleting consensus metadata
I20260929 18:58:50.039604 8404 tablet_bootstrap.cc:492] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: No bootstrap required, opened a new log
I20260929 18:58:50.039590 7877 catalog_manager.cc:4841] TS e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411): tablet e11918b9af86477a9b4737ab471371db (table test-table [id=0e4e2984953c4f26aeff08e8b72c09c2]) successfully deleted
I20260929 18:58:50.039664 8404 ts_tablet_manager.cc:1397] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 18:58:50.039767 8404 raft_consensus.cc:348] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.039819 8404 raft_consensus.cc:374] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.039831 8404 raft_consensus.cc:729] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.039891 8404 consensus_queue.cc:260] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.039978 8404 ts_tablet_manager.cc:1428] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 18:58:50.062180 8096 tablet.cc:2369] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:50.076598 8411 raft_consensus.cc:482] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:50.076696 8411 raft_consensus.cc:504] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.077039 8411 leader_election.cc:288] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.080389 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.077458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47767 (local address 127.0.98.3:42997)
0929 18:58:50.077526 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.077528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.077535 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:50.077560 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.077561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.077569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.077616 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.077803 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.078388 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.079622 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.080015 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.080107 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.080118 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.080279 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.080320 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:58:50.080405 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.077208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42997 (local address 127.0.98.2:47767)
0929 18:58:50.077318 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.077348 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.077389 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.077662 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.077663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.077664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.077724 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.077726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.078487 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.078488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.079542 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.079544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.080109 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.080110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.080180 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.080183 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.080272 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.080319 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 18:58:50.080694 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462daa5bccc498385e8ae802cb7a593" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" is_pre_election: true
I20260929 18:58:50.080796 8329 raft_consensus.cc:2454] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 0.
I20260929 18:58:50.080971 8122 leader_election.cc:302] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, 886a0f71db9f4ebaa254b78630b27a4e; no voters:
I20260929 18:58:50.081097 8411 raft_consensus.cc:2781] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:50.081167 8411 raft_consensus.cc:482] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:50.081198 8411 raft_consensus.cc:3037] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.081355 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.077289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45607 (local address 127.0.98.1:41411)
0929 18:58:50.077349 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.077351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.077457 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:58:50.077539 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.077541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.077550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.077625 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.078253 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.078934 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.080403 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.080970 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.081015 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.081024 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.081250 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.081285 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:50.081563 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.077213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41411 (local address 127.0.98.2:45607)
0929 18:58:50.077393 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.077403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.077481 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.077631 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.077632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.077633 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.077724 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.077726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.079034 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.079035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.080294 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.080297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.081084 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.081084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.081162 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.081165 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.081441 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.081481 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":40,"thread_start_us":91,"threads_started":1}
I20260929 18:58:50.081877 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462daa5bccc498385e8ae802cb7a593" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525" is_pre_election: true
I20260929 18:58:50.082010 8049 raft_consensus.cc:2454] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 0.
I20260929 18:58:50.082126 8411 raft_consensus.cc:504] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.082273 8411 leader_election.cc:288] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 election: Requested vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.082382 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462daa5bccc498385e8ae802cb7a593" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525"
I20260929 18:58:50.082473 8049 raft_consensus.cc:3037] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.082476 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462daa5bccc498385e8ae802cb7a593" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
I20260929 18:58:50.082535 8329 raft_consensus.cc:3037] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.083228 8329 raft_consensus.cc:2454] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 1.
I20260929 18:58:50.083228 8049 raft_consensus.cc:2454] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 1.
I20260929 18:58:50.083407 8121 leader_election.cc:302] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886a0f71db9f4ebaa254b78630b27a4e, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.083492 8411 raft_consensus.cc:2781] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:50.083616 8411 raft_consensus.cc:686] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 LEADER]: Becoming Leader. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Running, Role: LEADER
I20260929 18:58:50.083686 8411 consensus_queue.cc:237] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.084412 7891 catalog_manager.cc:5495] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "886a0f71db9f4ebaa254b78630b27a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:50.123855 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.116016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.0.1:37544)
0929 18:58:50.116210 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.116220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.116279 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.116446 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.116447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.116754 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:50.116802 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.116804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.117741 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.117742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.118258 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.118259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.118699 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.118699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.118739 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.119349 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:50.119354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:50.120846 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:50.122705 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:50.122711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:50.122713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:50.122955 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:50.123165 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:50.123166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:50.123166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:50.123227 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:50.123505 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:50.123510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:50.123550 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.123604 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.123754 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260929 18:58:50.123957 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.116114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.62:45025)
0929 18:58:50.116185 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.116187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.116317 (+ 130us) server_negotiation.cc:407] Connection header received
0929 18:58:50.116347 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.116348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.116359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.116436 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:50.116889 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.117654 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.118345 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.118619 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.118695 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.120940 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:50.120941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:50.120942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:50.120951 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:50.122616 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:50.123031 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:50.123033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:50.123033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:50.123096 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:50.123306 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:50.123307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:50.123307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:50.123439 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:50.123509 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.123654 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.123837 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":43}
I20260929 18:58:50.125990 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37544:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:58:50.126243 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:50.127759 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc7cb345b8344979dd5cc9c1b672e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70c62801231841f69f95f45830a40272]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:50.127820 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc7cb345b8344979dd5cc9c1b672e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70c62801231841f69f95f45830a40272]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:50.127934 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc7cb345b8344979dd5cc9c1b672e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.127972 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc7cb345b8344979dd5cc9c1b672e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.129215 8403 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: Bootstrap starting.
I20260929 18:58:50.129587 8403 tablet_bootstrap.cc:654] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.129570 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc7cb345b8344979dd5cc9c1b672e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70c62801231841f69f95f45830a40272]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:58:50.129698 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc7cb345b8344979dd5cc9c1b672e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.130172 8403 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: No bootstrap required, opened a new log
I20260929 18:58:50.130211 8403 ts_tablet_manager.cc:1397] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:50.130430 8403 raft_consensus.cc:348] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.130478 8403 raft_consensus.cc:374] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.130491 8403 raft_consensus.cc:729] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.130554 8403 consensus_queue.cc:260] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.130654 8403 ts_tablet_manager.cc:1428] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.130663 8404 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Bootstrap starting.
I20260929 18:58:50.130719 8411 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.130784 8411 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.131069 8404 tablet_bootstrap.cc:654] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.131806 8404 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: No bootstrap required, opened a new log
I20260929 18:58:50.131855 8404 ts_tablet_manager.cc:1397] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:50.131979 8404 raft_consensus.cc:348] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.132041 8404 raft_consensus.cc:374] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.132061 8404 raft_consensus.cc:729] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.132098 8404 consensus_queue.cc:260] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.132187 8404 ts_tablet_manager.cc:1428] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.132275 8409 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.132392 8409 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.132512 8405 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: Bootstrap starting.
I20260929 18:58:50.132952 8405 tablet_bootstrap.cc:654] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.133636 8405 tablet_bootstrap.cc:492] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: No bootstrap required, opened a new log
I20260929 18:58:50.133677 8405 ts_tablet_manager.cc:1397] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:50.133780 8405 raft_consensus.cc:348] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.133831 8405 raft_consensus.cc:374] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.133844 8405 raft_consensus.cc:729] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.133899 8405 consensus_queue.cc:260] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.134013 8405 ts_tablet_manager.cc:1428] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.134032 8413 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.134112 8413 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 18:58:50.169946 8376 tablet.cc:2369] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:58:50.258788 8409 raft_consensus.cc:482] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:50.258879 8409 raft_consensus.cc:504] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.259214 8409 leader_election.cc:288] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519)
I20260929 18:58:50.262379 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.259560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48873 (local address 127.0.98.1:41411)
0929 18:58:50.259618 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.259620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.259634 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:58:50.259668 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.259669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.259692 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.259761 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.259955 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.260567 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.261516 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.262048 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.262110 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.262119 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.262297 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.262325 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:58:50.262396 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.259450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41411 (local address 127.0.98.3:48873)
0929 18:58:50.259521 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.259534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.259610 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.259790 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.259791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.259793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.259875 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.259877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.260653 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.260654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.261426 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.261428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.262125 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.262126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.262198 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.262201 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.262286 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.262323 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:58:50.262632 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc7cb345b8344979dd5cc9c1b672e08" candidate_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525" is_pre_election: true
I20260929 18:58:50.262727 8049 raft_consensus.cc:2454] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3acc5252c50f4ad6855b1c52dd6ffe8c in term 0.
I20260929 18:58:50.262785 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.259593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43519 (local address 127.0.98.3:44097)
0929 18:58:50.259793 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.259804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.259870 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.260007 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.260008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.260009 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:58:50.260088 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.260090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.260712 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.260713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.261859 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.261862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.262541 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.262541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.262607 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.262611 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.262666 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.262699 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260929 18:58:50.262818 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.259660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44097 (local address 127.0.98.2:43519)
0929 18:58:50.259724 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.259726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.259870 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:58:50.259919 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.259920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.259930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.259984 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:58:50.260167 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.260627 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.261949 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.262436 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.262499 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.262509 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.262686 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.262735 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:58:50.262986 8261 leader_election.cc:302] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.263130 8409 raft_consensus.cc:2781] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:50.263168 8409 raft_consensus.cc:482] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:50.263195 8409 raft_consensus.cc:3037] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.263337 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc7cb345b8344979dd5cc9c1b672e08" candidate_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e" is_pre_election: true
I20260929 18:58:50.263435 8189 raft_consensus.cc:2454] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3acc5252c50f4ad6855b1c52dd6ffe8c in term 0.
I20260929 18:58:50.263999 8409 raft_consensus.cc:504] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.264124 8409 leader_election.cc:288] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [CANDIDATE]: Term 1 election: Requested vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519)
I20260929 18:58:50.264292 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc7cb345b8344979dd5cc9c1b672e08" candidate_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525"
I20260929 18:58:50.264382 8049 raft_consensus.cc:3037] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.264500 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc7cb345b8344979dd5cc9c1b672e08" candidate_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e"
I20260929 18:58:50.264578 8189 raft_consensus.cc:3037] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.265183 8049 raft_consensus.cc:2454] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3acc5252c50f4ad6855b1c52dd6ffe8c in term 1.
I20260929 18:58:50.265343 8261 leader_election.cc:302] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.265355 8189 raft_consensus.cc:2454] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3acc5252c50f4ad6855b1c52dd6ffe8c in term 1.
I20260929 18:58:50.265458 8409 raft_consensus.cc:2781] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:50.265626 8409 raft_consensus.cc:686] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 LEADER]: Becoming Leader. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Running, Role: LEADER
I20260929 18:58:50.265710 8409 consensus_queue.cc:237] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.266073 8444 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: TxnStatusTablet state changed. Reason: New leader 3acc5252c50f4ad6855b1c52dd6ffe8c. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.266227 8444 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:50.266440 8446 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:58:50.266546 8446 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:58:50.266474 7891 catalog_manager.cc:5495] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } health_report { overall_health: HEALTHY } } }
I20260929 18:58:50.281401 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.278610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42997 (local address 127.0.0.1:57652)
0929 18:58:50.278696 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.278708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.278775 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.278940 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.278940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.278941 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:50.278991 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.278993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.279890 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.279891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.280452 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.280454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.280913 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.280914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.280962 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.280976 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.281255 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.281262 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.281308 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.281345 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:58:50.281405 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.278677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.98.3:42997)
0929 18:58:50.278715 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.278718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.278745 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:58:50.278850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.278851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.278869 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.278940 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.279030 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.279805 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.280534 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.280836 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.280906 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.281071 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.281187 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.281260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.281313 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.281331 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:50.282591 8189 raft_consensus.cc:1261] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Refusing update from remote peer 3acc5252c50f4ad6855b1c52dd6ffe8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.282640 8049 raft_consensus.cc:1261] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Refusing update from remote peer 3acc5252c50f4ad6855b1c52dd6ffe8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.282860 8444 consensus_queue.cc:1035] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.282969 8409 consensus_queue.cc:1035] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.283520 8413 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: TxnStatusTablet state changed. Reason: New leader 3acc5252c50f4ad6855b1c52dd6ffe8c. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.283528 8411 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: TxnStatusTablet state changed. Reason: New leader 3acc5252c50f4ad6855b1c52dd6ffe8c. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.283614 8413 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.283622 8411 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.284389 8444 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.284474 8444 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:50.284534 8417 mvcc.cc:205] Tried to move back new op lower bound from 7334741320835473408 to 7334741320768880640. Current Snapshot: MvccSnapshot[applied={T|T < 7334741320835473408}]
I20260929 18:58:50.284631 8411 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.284690 8411 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 886a0f71db9f4ebaa254b78630b27a4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.284814 8444 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.284888 8444 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P 3acc5252c50f4ad6855b1c52dd6ffe8c: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:58:50.285548 8413 tablet_replica.cc:439] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } } }
I20260929 18:58:50.285636 8413 tablet_replica.cc:442] T 7bc7cb345b8344979dd5cc9c1b672e08 P e2ebfad725ef479cb9d833f2bf96a525: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:58:50.289600 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.287064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.0.98.2:43519)
0929 18:58:50.287102 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.287104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.287113 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:50.287178 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.287179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.287196 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.287272 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.287419 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.288140 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.288831 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.289130 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.289198 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.289301 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.289416 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.289482 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.289503 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.289519 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:58:50.294744 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.292040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41943 (local address 127.0.98.3:42997)
0929 18:58:50.292100 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.292102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.292249 (+ 147us) server_negotiation.cc:407] Connection header received
0929 18:58:50.292313 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.292314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.292322 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.292394 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.292546 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.292992 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.294059 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.294277 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.294371 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.294543 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.294568 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.294607 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.294685 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.294699 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:58:50.294814 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.291951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42997 (local address 127.0.98.2:41943)
0929 18:58:50.292173 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.292184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.292257 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.292406 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.292407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.292409 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:50.292472 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.292474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.293062 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.293063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.293978 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.293980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.294384 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.294386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.294451 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.294464 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.294630 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.294632 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.294684 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.294727 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
I20260929 18:58:50.297366 8329 raft_consensus.cc:1261] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Refusing update from remote peer 886a0f71db9f4ebaa254b78630b27a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.297366 8049 raft_consensus.cc:1261] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Refusing update from remote peer 886a0f71db9f4ebaa254b78630b27a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.297580 8411 consensus_queue.cc:1035] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.297736 8432 consensus_queue.cc:1035] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.306207 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.303510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43519 (local address 127.0.98.3:57007)
0929 18:58:50.303567 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.303579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:50.303619 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:50.303747 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:50.303747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:50.303749 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:58:50.303795 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.303796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.304690 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.304691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.305408 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:50.305411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.305816 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.305817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:50.305884 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.305896 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.306057 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.306059 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:58:50.306103 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:58:50.306142 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 18:58:50.306267 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.303599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57007 (local address 127.0.98.2:43519)
0929 18:58:50.303641 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.303642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.303651 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:50.303684 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.303684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.303693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.303739 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.303864 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.304597 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.305491 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.305733 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.305805 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.305979 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.306011 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.306059 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.306114 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.306132 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:58:50.316180 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.316298 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.316917 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Sending DeleteTablet for 3 replicas of tablet 1462daa5bccc498385e8ae802cb7a593
I20260929 18:58:50.317314 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462daa5bccc498385e8ae802cb7a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 18:58:50.317430 8426 tablet_replica.cc:330] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: stopping tablet replica
I20260929 18:58:50.317509 8426 raft_consensus.cc:2229] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.317546 8426 raft_consensus.cc:2258] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.317992 8426 ts_tablet_manager.cc:1905] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.319311 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462daa5bccc498385e8ae802cb7a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57638
I20260929 18:58:50.319518 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462daa5bccc498385e8ae802cb7a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47618
I20260929 18:58:50.319625 8427 tablet_replica.cc:330] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: stopping tablet replica
I20260929 18:58:50.319679 8427 raft_consensus.cc:2229] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:50.319680 8425 tablet_replica.cc:330] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: stopping tablet replica
I20260929 18:58:50.319741 8425 raft_consensus.cc:2229] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.319777 8425 raft_consensus.cc:2258] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.319770 8427 raft_consensus.cc:2258] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.320099 8427 ts_tablet_manager.cc:1905] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.321633 8427 ts_tablet_manager.cc:1918] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:50.321695 8427 log.cc:1196] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/1462daa5bccc498385e8ae802cb7a593
I20260929 18:58:50.321913 8427 ts_tablet_manager.cc:1939] T 1462daa5bccc498385e8ae802cb7a593 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting consensus metadata
I20260929 18:58:50.322283 8426 ts_tablet_manager.cc:1918] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:50.322337 8426 log.cc:1196] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/1462daa5bccc498385e8ae802cb7a593
I20260929 18:58:50.322341 7877 catalog_manager.cc:4841] TS 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519): tablet 1462daa5bccc498385e8ae802cb7a593 (table test-table [id=0325fd2a2d4947bbbdb1376bde732e35]) successfully deleted
I20260929 18:58:50.322551 8426 ts_tablet_manager.cc:1939] T 1462daa5bccc498385e8ae802cb7a593 P e2ebfad725ef479cb9d833f2bf96a525: Deleting consensus metadata
I20260929 18:58:50.322852 7877 catalog_manager.cc:4841] TS e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411): tablet 1462daa5bccc498385e8ae802cb7a593 (table test-table [id=0325fd2a2d4947bbbdb1376bde732e35]) successfully deleted
I20260929 18:58:50.327539 8425 ts_tablet_manager.cc:1905] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.329440 8425 ts_tablet_manager.cc:1918] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:50.329491 8425 log.cc:1196] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/1462daa5bccc498385e8ae802cb7a593
I20260929 18:58:50.329705 8425 ts_tablet_manager.cc:1939] T 1462daa5bccc498385e8ae802cb7a593 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting consensus metadata
I20260929 18:58:50.330227 7878 catalog_manager.cc:4841] TS 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997): tablet 1462daa5bccc498385e8ae802cb7a593 (table test-table [id=0325fd2a2d4947bbbdb1376bde732e35]) successfully deleted
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:58:50.350236 7942 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:50.344436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.62:45025)
0929 18:58:50.344488 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.344490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.344652 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:58:50.344737 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.344738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.344749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.344824 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:50.345395 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.345893 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.346712 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.346968 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.347048 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.350168 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37556: BlockingRecv error: recv got EOF from 127.0.0.1:37556 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:58:50.363559 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.351388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.98.62:45025)
0929 18:58:50.351426 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.351428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.355505 (+ 4077us) server_negotiation.cc:407] Connection header received
0929 18:58:50.355593 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.355594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.355604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.355675 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:50.356213 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.356896 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.357931 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.358223 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.358264 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.360879 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:50.360881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:50.360881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:50.360888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:50.362356 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:50.362719 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:50.362720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:50.362721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:50.362788 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:50.362990 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:50.362991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:50.362991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:50.363108 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:50.363218 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.363367 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.363451 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:58:50.365154 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:50.365334 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:50.366720 8309 tablet_service.cc:1467] Processing CreateTablet for tablet ddb0e6217be44c05bf50af37f118a630 (DEFAULT_TABLE table=test-table [id=5722d08e372e404c8e13a254bb7a1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.366779 8029 tablet_service.cc:1467] Processing CreateTablet for tablet ddb0e6217be44c05bf50af37f118a630 (DEFAULT_TABLE table=test-table [id=5722d08e372e404c8e13a254bb7a1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.366859 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb0e6217be44c05bf50af37f118a630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.366879 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb0e6217be44c05bf50af37f118a630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.368196 8405 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Bootstrap starting.
I20260929 18:58:50.368192 8404 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Bootstrap starting.
I20260929 18:58:50.368228 8169 tablet_service.cc:1467] Processing CreateTablet for tablet ddb0e6217be44c05bf50af37f118a630 (DEFAULT_TABLE table=test-table [id=5722d08e372e404c8e13a254bb7a1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.368337 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb0e6217be44c05bf50af37f118a630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.368510 8405 tablet_bootstrap.cc:654] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.368639 8404 tablet_bootstrap.cc:654] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.369575 8403 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Bootstrap starting.
I20260929 18:58:50.369962 8403 tablet_bootstrap.cc:654] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.370580 8403 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: No bootstrap required, opened a new log
I20260929 18:58:50.370643 8403 ts_tablet_manager.cc:1397] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:50.370813 8403 raft_consensus.cc:348] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.370890 8403 raft_consensus.cc:374] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.370918 8403 raft_consensus.cc:729] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.370954 8403 consensus_queue.cc:260] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.371042 8403 ts_tablet_manager.cc:1428] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.375687 8405 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: No bootstrap required, opened a new log
I20260929 18:58:50.375756 8405 ts_tablet_manager.cc:1397] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 18:58:50.375869 8405 raft_consensus.cc:348] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.375917 8405 raft_consensus.cc:374] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.375929 8405 raft_consensus.cc:729] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.375983 8405 consensus_queue.cc:260] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.376070 8405 ts_tablet_manager.cc:1428] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.380235 8404 tablet_bootstrap.cc:492] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: No bootstrap required, opened a new log
I20260929 18:58:50.380307 8404 ts_tablet_manager.cc:1397] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260929 18:58:50.380470 8404 raft_consensus.cc:348] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.380518 8404 raft_consensus.cc:374] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.380532 8404 raft_consensus.cc:729] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.380586 8404 consensus_queue.cc:260] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.380702 8404 ts_tablet_manager.cc:1428] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.407263 8413 raft_consensus.cc:482] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:50.407436 8413 raft_consensus.cc:504] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.407598 8413 leader_election.cc:288] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.407783 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb0e6217be44c05bf50af37f118a630" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" is_pre_election: true
I20260929 18:58:50.407783 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb0e6217be44c05bf50af37f118a630" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e" is_pre_election: true
I20260929 18:58:50.407866 8189 raft_consensus.cc:2454] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 0.
I20260929 18:58:50.407866 8329 raft_consensus.cc:2454] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 0.
I20260929 18:58:50.408017 7982 leader_election.cc:302] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.408136 8413 raft_consensus.cc:2781] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:50.408170 8413 raft_consensus.cc:482] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:50.408190 8413 raft_consensus.cc:3037] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.408946 8413 raft_consensus.cc:504] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.409101 8413 leader_election.cc:288] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 election: Requested vote from peers 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.409289 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb0e6217be44c05bf50af37f118a630" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886a0f71db9f4ebaa254b78630b27a4e"
I20260929 18:58:50.409373 8189 raft_consensus.cc:3037] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.409385 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb0e6217be44c05bf50af37f118a630" candidate_uuid: "e2ebfad725ef479cb9d833f2bf96a525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
I20260929 18:58:50.409444 8329 raft_consensus.cc:3037] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.410140 8189 raft_consensus.cc:2454] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 1.
I20260929 18:58:50.410140 8329 raft_consensus.cc:2454] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ebfad725ef479cb9d833f2bf96a525 in term 1.
I20260929 18:58:50.410311 7982 leader_election.cc:302] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.410413 8413 raft_consensus.cc:2781] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:50.410451 8413 raft_consensus.cc:686] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 LEADER]: Becoming Leader. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Running, Role: LEADER
I20260929 18:58:50.410498 8413 consensus_queue.cc:237] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.410995 7892 catalog_manager.cc:5495] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2ebfad725ef479cb9d833f2bf96a525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:50.417227 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.414772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.1:41411)
0929 18:58:50.414816 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.414818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.414860 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:58:50.414927 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.414928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.414938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.414993 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.415125 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.415723 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.416510 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.416722 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.416794 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.416960 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.416993 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.417062 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.417122 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.417143 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:58:50.417886 8189 raft_consensus.cc:1261] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Refusing update from remote peer e2ebfad725ef479cb9d833f2bf96a525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.417881 8329 raft_consensus.cc:1261] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Refusing update from remote peer e2ebfad725ef479cb9d833f2bf96a525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.418071 8410 consensus_queue.cc:1035] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.418159 8413 consensus_queue.cc:1035] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.421029 7890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.421150 7890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.422111 7890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Sending DeleteTablet for 3 replicas of tablet ddb0e6217be44c05bf50af37f118a630
I20260929 18:58:50.422395 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb0e6217be44c05bf50af37f118a630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57638
I20260929 18:58:50.422402 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb0e6217be44c05bf50af37f118a630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47618
I20260929 18:58:50.422477 8427 tablet_replica.cc:330] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: stopping tablet replica
I20260929 18:58:50.422523 8427 raft_consensus.cc:2229] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.422567 8427 raft_consensus.cc:2258] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.422627 8425 tablet_replica.cc:330] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: stopping tablet replica
I20260929 18:58:50.422673 8425 raft_consensus.cc:2229] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.422716 8425 raft_consensus.cc:2258] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.423077 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:58:50.423216 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:58:50.423303 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb0e6217be44c05bf50af37f118a630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 18:58:50.423326 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb0e6217be44c05bf50af37f118a630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 18:58:50.423377 8426 tablet_replica.cc:330] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: stopping tablet replica
I20260929 18:58:50.423441 8426 raft_consensus.cc:2229] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:50.423499 8426 raft_consensus.cc:2258] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.423983 8426 ts_tablet_manager.cc:1905] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.423983 8427 ts_tablet_manager.cc:1905] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.425541 8425 ts_tablet_manager.cc:1905] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.426342 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a (DEFAULT_TABLE table=test-table [id=6f2e17e7288e478e8f91852e3003dbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.426517 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a (DEFAULT_TABLE table=test-table [id=6f2e17e7288e478e8f91852e3003dbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.426600 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c3c7a09034836a64f67211bce3e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.427089 8425 ts_tablet_manager.cc:1918] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.427140 8425 log.cc:1196] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/ddb0e6217be44c05bf50af37f118a630
I20260929 18:58:50.427474 8427 ts_tablet_manager.cc:1918] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.427524 8427 log.cc:1196] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/ddb0e6217be44c05bf50af37f118a630
I20260929 18:58:50.427747 8427 ts_tablet_manager.cc:1939] T ddb0e6217be44c05bf50af37f118a630 P 886a0f71db9f4ebaa254b78630b27a4e: Deleting consensus metadata
I20260929 18:58:50.427870 8404 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Bootstrap starting.
I20260929 18:58:50.427888 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a (DEFAULT_TABLE table=test-table [id=6f2e17e7288e478e8f91852e3003dbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:58:50.428265 8404 tablet_bootstrap.cc:654] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.428580 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c3c7a09034836a64f67211bce3e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.429421 8403 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Bootstrap starting.
I20260929 18:58:50.429795 8404 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: No bootstrap required, opened a new log
I20260929 18:58:50.429836 8403 tablet_bootstrap.cc:654] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.429842 8404 ts_tablet_manager.cc:1397] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:58:50.429996 8404 raft_consensus.cc:348] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.430570 8404 raft_consensus.cc:374] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.430595 8404 raft_consensus.cc:729] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acc5252c50f4ad6855b1c52dd6ffe8c, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.430658 8404 consensus_queue.cc:260] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.430776 8404 ts_tablet_manager.cc:1428] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:58:50.431244 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c3c7a09034836a64f67211bce3e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:50.432593 8405 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: Bootstrap starting.
I20260929 18:58:50.431003 8426 ts_tablet_manager.cc:1918] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:58:50.432663 8426 log.cc:1196] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/ddb0e6217be44c05bf50af37f118a630
I20260929 18:58:50.432875 8405 tablet_bootstrap.cc:654] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:50.433173 7877 catalog_manager.cc:4841] TS 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519): tablet ddb0e6217be44c05bf50af37f118a630 (table test-table [id=5722d08e372e404c8e13a254bb7a1bd8]) successfully deleted
I20260929 18:58:50.433357 8425 ts_tablet_manager.cc:1939] T ddb0e6217be44c05bf50af37f118a630 P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting consensus metadata
I20260929 18:58:50.432888 8426 ts_tablet_manager.cc:1939] T ddb0e6217be44c05bf50af37f118a630 P e2ebfad725ef479cb9d833f2bf96a525: Deleting consensus metadata
I20260929 18:58:50.433684 7878 catalog_manager.cc:4841] TS 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997): tablet ddb0e6217be44c05bf50af37f118a630 (table test-table [id=5722d08e372e404c8e13a254bb7a1bd8]) successfully deleted
I20260929 18:58:50.433931 7877 catalog_manager.cc:4841] TS e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411): tablet ddb0e6217be44c05bf50af37f118a630 (table test-table [id=5722d08e372e404c8e13a254bb7a1bd8]) successfully deleted
W20260929 18:58:50.434047 7877 catalog_manager.cc:4808] TS e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411): delete failed for tablet ddb0e6217be44c05bf50af37f118a630 because the tablet was not found. No further retry: Not found: Tablet not found: ddb0e6217be44c05bf50af37f118a630
I20260929 18:58:50.434453 8405 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: No bootstrap required, opened a new log
I20260929 18:58:50.434504 8405 ts_tablet_manager.cc:1397] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:58:50.434603 8405 raft_consensus.cc:348] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.434662 8405 raft_consensus.cc:374] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.434677 8405 raft_consensus.cc:729] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebfad725ef479cb9d833f2bf96a525, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.434733 8405 consensus_queue.cc:260] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.434819 8405 ts_tablet_manager.cc:1428] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.435392 8403 tablet_bootstrap.cc:492] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: No bootstrap required, opened a new log
I20260929 18:58:50.435444 8403 ts_tablet_manager.cc:1397] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260929 18:58:50.435591 8403 raft_consensus.cc:348] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.435639 8403 raft_consensus.cc:374] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:50.435655 8403 raft_consensus.cc:729] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Initialized, Role: FOLLOWER
I20260929 18:58:50.435711 8403 consensus_queue.cc:260] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.435798 8403 ts_tablet_manager.cc:1428] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:50.500984 8432 raft_consensus.cc:482] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:58:50.501164 8432 raft_consensus.cc:504] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.501385 8432 leader_election.cc:288] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.501618 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c3c7a09034836a64f67211bce3e2a" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525" is_pre_election: true
I20260929 18:58:50.501646 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c3c7a09034836a64f67211bce3e2a" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" is_pre_election: true
I20260929 18:58:50.501752 8049 raft_consensus.cc:2454] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 0.
I20260929 18:58:50.501762 8329 raft_consensus.cc:2454] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 0.
I20260929 18:58:50.501925 8121 leader_election.cc:302] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886a0f71db9f4ebaa254b78630b27a4e, e2ebfad725ef479cb9d833f2bf96a525; no voters:
I20260929 18:58:50.502041 8432 raft_consensus.cc:2781] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:58:50.502094 8432 raft_consensus.cc:482] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:58:50.502110 8432 raft_consensus.cc:3037] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.502839 8432 raft_consensus.cc:504] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.502974 8432 leader_election.cc:288] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 election: Requested vote from peers e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411), 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997)
I20260929 18:58:50.503129 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c3c7a09034836a64f67211bce3e2a" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c"
I20260929 18:58:50.503235 8329 raft_consensus.cc:3037] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.503208 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c3c7a09034836a64f67211bce3e2a" candidate_uuid: "886a0f71db9f4ebaa254b78630b27a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ebfad725ef479cb9d833f2bf96a525"
I20260929 18:58:50.503288 8049 raft_consensus.cc:3037] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:50.503859 8329 raft_consensus.cc:2454] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 1.
I20260929 18:58:50.503965 8049 raft_consensus.cc:2454] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886a0f71db9f4ebaa254b78630b27a4e in term 1.
I20260929 18:58:50.504076 8122 leader_election.cc:302] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acc5252c50f4ad6855b1c52dd6ffe8c, 886a0f71db9f4ebaa254b78630b27a4e; no voters:
I20260929 18:58:50.504186 8432 raft_consensus.cc:2781] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:50.504271 8432 raft_consensus.cc:686] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 1 LEADER]: Becoming Leader. State: Replica: 886a0f71db9f4ebaa254b78630b27a4e, State: Running, Role: LEADER
I20260929 18:58:50.504343 8432 consensus_queue.cc:237] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } }
I20260929 18:58:50.504911 7892 catalog_manager.cc:5495] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "886a0f71db9f4ebaa254b78630b27a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886a0f71db9f4ebaa254b78630b27a4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 } health_report { overall_health: UNKNOWN } } }
I20260929 18:58:50.525851 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.523794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.0.98.2:43519)
0929 18:58:50.523833 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:50.523835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:50.523842 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:50.523911 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:50.523912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:50.523919 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:50.523966 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:58:50.524106 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.524561 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.525315 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:50.525490 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:50.525545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:50.525642 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:58:50.525666 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:58:50.525699 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:50.525730 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 18:58:50.525746 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:58:50.527869 8049 raft_consensus.cc:1261] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Refusing update from remote peer 886a0f71db9f4ebaa254b78630b27a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.527897 8329 raft_consensus.cc:1261] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Refusing update from remote peer 886a0f71db9f4ebaa254b78630b27a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:58:50.528095 8432 consensus_queue.cc:1035] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ebfad725ef479cb9d833f2bf96a525" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.528213 8411 consensus_queue.cc:1035] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acc5252c50f4ad6855b1c52dd6ffe8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:58:50.537755 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.537855 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:58:50.538432 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2656c285ea82433a9fac550f47496a29: Sending DeleteTablet for 3 replicas of tablet 5f6c3c7a09034836a64f67211bce3e2a
I20260929 18:58:50.538712 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 18:58:50.538734 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47618
I20260929 18:58:50.538754 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c3c7a09034836a64f67211bce3e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:58:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57638
I20260929 18:58:50.538851 8426 tablet_replica.cc:330] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525: stopping tablet replica
I20260929 18:58:50.538921 8426 raft_consensus.cc:2229] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.538965 8426 raft_consensus.cc:2258] T 5f6c3c7a09034836a64f67211bce3e2a P e2ebfad725ef479cb9d833f2bf96a525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.538993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7964
I20260929 18:58:50.539043 8427 tablet_replica.cc:330] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: stopping tablet replica
I20260929 18:58:50.539108 8427 raft_consensus.cc:2229] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:58:50.539162 8427 raft_consensus.cc:2258] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.539232 8425 tablet_replica.cc:330] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: stopping tablet replica
I20260929 18:58:50.539305 8425 raft_consensus.cc:2229] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:58:50.539340 8425 raft_consensus.cc:2258] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:58:50.540191 8427 ts_tablet_manager.cc:1905] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.540356 8425 ts_tablet_manager.cc:1905] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:58:50.541754 8427 ts_tablet_manager.cc:1918] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:50.541827 8427 log.cc:1196] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/5f6c3c7a09034836a64f67211bce3e2a
I20260929 18:58:50.542021 8427 ts_tablet_manager.cc:1939] T 5f6c3c7a09034836a64f67211bce3e2a P 886a0f71db9f4ebaa254b78630b27a4e: Deleting consensus metadata
I20260929 18:58:50.542141 8425 ts_tablet_manager.cc:1918] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:58:50.542194 8425 log.cc:1196] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/5f6c3c7a09034836a64f67211bce3e2a
I20260929 18:58:50.542302 7877 catalog_manager.cc:4841] TS 886a0f71db9f4ebaa254b78630b27a4e (127.0.98.2:43519): tablet 5f6c3c7a09034836a64f67211bce3e2a (table test-table [id=6f2e17e7288e478e8f91852e3003dbb4]) successfully deleted
I20260929 18:58:50.542407 8425 ts_tablet_manager.cc:1939] T 5f6c3c7a09034836a64f67211bce3e2a P 3acc5252c50f4ad6855b1c52dd6ffe8c: Deleting consensus metadata
I20260929 18:58:50.542719 7878 catalog_manager.cc:4841] TS 3acc5252c50f4ad6855b1c52dd6ffe8c (127.0.98.3:42997): tablet 5f6c3c7a09034836a64f67211bce3e2a (table test-table [id=6f2e17e7288e478e8f91852e3003dbb4]) successfully deleted
W20260929 18:58:50.546903 7877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41411 (error 108)
I20260929 18:58:50.547188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8104
I20260929 18:58:50.547868 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:50.547270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41411 (local address 127.0.0.1:55330)
0929 18:58:50.547768 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:50.547820 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":57,"threads_started":1}
W20260929 18:58:50.548132 7877 catalog_manager.cc:4584] TS e2ebfad725ef479cb9d833f2bf96a525 (127.0.98.1:41411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f6c3c7a09034836a64f67211bce3e2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41411: connect: Connection refused (error 111)
I20260929 18:58:50.552953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8244
I20260929 18:58:50.558764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 7860
2026-09-29T18:58:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2468 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[8490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[8490](info): set up 2 sockets
Sep 29 18:58:50 dist-test-slave-gfsd krb5kdc[8490](info): commencing operation
krb5kdc: starting...
W20260929 18:58:51.599857 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:51 dist-test-slave-gfsd krb5kdc[8490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:51.716056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34257
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:51.795629 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:51.795830 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:51.795851 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:51.796988 8503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:51.797029 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:51.797042 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:51.797055 8503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:51.797107 8503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:51.798584 8503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:51.798830 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:51.799062 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:58:51.804076 8503 server_base.cc:1034] running on GCE node
W20260929 18:58:51.804059 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:51.804087 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:51.804055 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:51.804533 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:51.804826 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:51.805976 8503 hybrid_clock.cc:648] HybridClock initialized: now 1790708331805955 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:58:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1254 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:58:51 dist-test-slave-gfsd krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:58:51 dist-test-slave-gfsd krb5kdc[8516](info): set up 2 sockets
Sep 29 18:58:51 dist-test-slave-gfsd krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260929 18:58:52.851943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:58:52 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:58:52.970719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:53.055769 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.056006 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.056042 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.057204 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:53.057266 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.057291 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.057320 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:53.057353 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:53.058848 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.059124 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.059377 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.064447 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.064451 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.064494 8529 server_base.cc:1034] running on GCE node
W20260929 18:58:53.064486 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.064981 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.065284 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.066444 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790708333066427 us; error 29 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.072533 8529 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:53.073060 8529 webserver.cc:460] Webserver started at http://127.0.98.62:46295/ using document root <none> and password file <none>
I20260929 18:58:53.073302 8529 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.073371 8529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.073529 8529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:53.074651 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "80a590b06ef14017879f65e2d1dbb772"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "c2b8f28b6092f41778db7c7b0e247e81"
server_key_iv: "7c48f0598ac20bbbd5aa48d290ec028c"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.075052 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "80a590b06ef14017879f65e2d1dbb772"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "c2b8f28b6092f41778db7c7b0e247e81"
server_key_iv: "7c48f0598ac20bbbd5aa48d290ec028c"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.076508 8529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:58:53.077291 8544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.077484 8529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:53.077592 8529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "80a590b06ef14017879f65e2d1dbb772"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "c2b8f28b6092f41778db7c7b0e247e81"
server_key_iv: "7c48f0598ac20bbbd5aa48d290ec028c"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.077666 8529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.089244 8529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.091683 8529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.091866 8529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.095754 8529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37625
I20260929 18:58:53.095796 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37625 every 8 connection(s)
I20260929 18:58:53.096313 8529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:53.097008 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:58:53.099434 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Bootstrap starting.
I20260929 18:58:53.100054 8597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Neither blocks nor log segments found. Creating new log.
I20260929 18:58:53.100370 8597 log.cc:824] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:53.101055 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: No bootstrap required, opened a new log
I20260929 18:58:53.102749 8597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.102898 8597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:58:53.102926 8597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a590b06ef14017879f65e2d1dbb772, State: Initialized, Role: FOLLOWER
I20260929 18:58:53.103116 8597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.103188 8597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:53.103212 8597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:53.103310 8597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:58:53.104221 8597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.104350 8597 leader_election.cc:302] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a590b06ef14017879f65e2d1dbb772; no voters:
I20260929 18:58:53.104544 8597 leader_election.cc:288] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:58:53.104615 8600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:58:53.104779 8600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 LEADER]: Becoming Leader. State: Replica: 80a590b06ef14017879f65e2d1dbb772, State: Running, Role: LEADER
I20260929 18:58:53.104821 8597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:53.104921 8600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.105273 8601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80a590b06ef14017879f65e2d1dbb772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260929 18:58:53.105376 8601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:53.105278 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80a590b06ef14017879f65e2d1dbb772. Latest consensus state: current_term: 1 leader_uuid: "80a590b06ef14017879f65e2d1dbb772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260929 18:58:53.105504 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:53.105604 8607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:53.105605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8529
I20260929 18:58:53.105741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:58:53.105988 392 external_mini_cluster.cc:1428] Setting key e892d8a14ab8de3d52f15651240e54ab
I20260929 18:58:53.106128 8607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:53.108091 8607 catalog_manager.cc:1347] Generated new cluster ID: 3ae1d6c95d514137a943399ae3df44f4
I20260929 18:58:53.108144 8607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:53.119772 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.106717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.0.98.62:37625)
0929 18:58:53.106928 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.106938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.107058 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:58:53.107212 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.107219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.107300 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.107450 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.111555 (+ 4105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.112584 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.113383 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.113837 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.113959 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.116475 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:53.116479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:53.116480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:53.116492 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:53.118182 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.118559 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.118562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.118562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.118651 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.118886 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.118887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.118887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.119124 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:53.119337 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:53.119407 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:58:53.119546 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260929 18:58:53.121105 8607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:58:53.121691 8607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:53.124573 8607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Generated new TSK 0
I20260929 18:58:53.124749 8607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:58:53.158952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:53.244338 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.244530 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.244553 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.245702 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.245743 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.245769 8623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:53.247267 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.247570 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.247812 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.252832 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.252900 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.252918 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.253504 8623 server_base.cc:1034] running on GCE node
I20260929 18:58:53.253743 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.254027 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.255249 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790708333255238 us; error 31 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.258312 8623 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:58:53.258810 8623 webserver.cc:460] Webserver started at http://127.0.98.1:46353/ using document root <none> and password file <none>
I20260929 18:58:53.259088 8623 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.259181 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.259387 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:53.260627 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "57da0d85214c45bb8adddacbe9d8a12b"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "29ce22cf8f6f910765078509f8e03886"
server_key_iv: "1cb5294baf585c5caa7875d3765e2fa9"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.261049 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57da0d85214c45bb8adddacbe9d8a12b"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "29ce22cf8f6f910765078509f8e03886"
server_key_iv: "1cb5294baf585c5caa7875d3765e2fa9"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.262591 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:53.263408 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.263581 8623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.263674 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "57da0d85214c45bb8adddacbe9d8a12b"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "29ce22cf8f6f910765078509f8e03886"
server_key_iv: "1cb5294baf585c5caa7875d3765e2fa9"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.263751 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.274871 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.276760 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.276917 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.277148 8623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:53.277452 8623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:53.277506 8623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.277546 8623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:53.277575 8623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.283488 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42599
I20260929 18:58:53.283526 8751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42599 every 8 connection(s)
I20260929 18:58:53.283864 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:53.284758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8623
I20260929 18:58:53.284900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:58:53.285101 392 external_mini_cluster.cc:1428] Setting key 03e408e5a545bb2d4f2daf23d2ca12ac
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:58:53.294250 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.284639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56447 (local address 127.0.98.62:37625)
0929 18:58:53.284691 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.284693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.284940 (+ 247us) server_negotiation.cc:407] Connection header received
0929 18:58:53.286097 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.286098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.286115 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.286276 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.286846 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.287545 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.288825 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.289053 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.289138 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.291232 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:53.291233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:53.291234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:53.291245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:53.292754 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.293274 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.293275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.293275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.293336 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.293587 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.293588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.293588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.293717 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:53.293791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:53.294059 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 18:58:53.294148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:58:53.294481 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.284627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:56447)
0929 18:58:53.284790 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.284837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.286015 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.286326 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.286329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.286565 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.286754 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.286758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.287628 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.287630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.288731 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.288734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.289147 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.289148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.289272 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.289795 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.289803 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.291127 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:53.292855 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.292862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.292864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.293105 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.293404 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.293405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.293405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.293519 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.293782 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:53.293788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:53.293927 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:58:53.294037 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:58:53.294238 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260929 18:58:53.294989 8752 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260929 18:58:53.295070 8752 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:53.295205 8752 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
I20260929 18:58:53.295719 8561 ts_manager.cc:194] Registered new tserver with Master: 57da0d85214c45bb8adddacbe9d8a12b (127.0.98.1:42599)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:58:53.296591 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:58:53.312126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:53.392135 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.392370 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.392408 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.393548 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.393615 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.393661 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:53.395144 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.395423 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.395648 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.400640 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.400663 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.400640 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.400856 8757 server_base.cc:1034] running on GCE node
I20260929 18:58:53.401103 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.401350 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.402539 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790708333402533 us; error 37 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.405139 8757 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:58:53.405761 8757 webserver.cc:460] Webserver started at http://127.0.98.2:40195/ using document root <none> and password file <none>
I20260929 18:58:53.406006 8757 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.406071 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.406229 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:53.407394 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "e96e9a90e01c4a84abe614f0bfa9a890"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "b8fe064a6b37fbaaffafbcf2e0bce3f7"
server_key_iv: "fcfe1fe50d97fd996e01c89fac0f0aca"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.407749 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e96e9a90e01c4a84abe614f0bfa9a890"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "b8fe064a6b37fbaaffafbcf2e0bce3f7"
server_key_iv: "fcfe1fe50d97fd996e01c89fac0f0aca"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.409024 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 18:58:53.409801 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.409961 8757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.410063 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "e96e9a90e01c4a84abe614f0bfa9a890"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "b8fe064a6b37fbaaffafbcf2e0bce3f7"
server_key_iv: "fcfe1fe50d97fd996e01c89fac0f0aca"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.410135 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.433297 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.435350 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.435518 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.435760 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:53.436092 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:53.436179 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.436244 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:53.436276 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.442193 8757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43473
I20260929 18:58:53.442241 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43473 every 8 connection(s)
I20260929 18:58:53.442548 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:53.446985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8757
I20260929 18:58:53.447206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:58:53.447469 392 external_mini_cluster.cc:1428] Setting key 92d42c60411dd180d58596d8ca96c9dd
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:53.452765 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.443268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34361 (local address 127.0.98.62:37625)
0929 18:58:53.443332 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.443334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.443550 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:58:53.444557 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.444559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.444572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.444619 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.445259 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.445788 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.446816 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.447012 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.447096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.449772 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:53.449774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:53.449774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:53.449782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:53.451218 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.451796 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.451796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.451797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.451847 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.452135 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.452136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.452137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.452236 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:53.452316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:53.452643 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 18:58:53.452703 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:58:53.453078 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.443226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:34361)
0929 18:58:53.443429 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.443466 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.444431 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.444688 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.444693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.444961 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.445154 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.445158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.445887 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.445888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.446711 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.446713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.447145 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.447147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.447335 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.447966 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.447974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.449663 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:53.451388 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.451395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.451397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.451724 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.451911 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.451912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.451913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.452059 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.452333 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:53.452340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:53.452503 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 18:58:53.452624 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:58:53.452812 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260929 18:58:53.453622 8886 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260929 18:58:53.453737 8886 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:53.453943 8886 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
I20260929 18:58:53.454329 8561 ts_manager.cc:194] Registered new tserver with Master: e96e9a90e01c4a84abe614f0bfa9a890 (127.0.98.2:43473)
I20260929 18:58:53.454704 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34361
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:58:53.474722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:53.554279 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.554477 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.554500 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.555773 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.555817 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.555845 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:53.557315 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.557554 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.557796 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.562853 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.562876 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.562891 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.563107 8891 server_base.cc:1034] running on GCE node
I20260929 18:58:53.563442 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.563637 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.564783 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790708333564764 us; error 38 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.567500 8891 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:58:53.567924 8891 webserver.cc:460] Webserver started at http://127.0.98.3:44575/ using document root <none> and password file <none>
I20260929 18:58:53.568172 8891 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.568218 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.568341 8891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:58:53.569370 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "81b7604485744e059054ddfd7319580a"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "143d4671a2e117b2ba790785c8c58599"
server_key_iv: "b05906db813e8aad548003d37af12b81"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.569732 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81b7604485744e059054ddfd7319580a"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "143d4671a2e117b2ba790785c8c58599"
server_key_iv: "b05906db813e8aad548003d37af12b81"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.571076 8891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:58:53.572059 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.572278 8891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:58:53.572399 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "81b7604485744e059054ddfd7319580a"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "143d4671a2e117b2ba790785c8c58599"
server_key_iv: "b05906db813e8aad548003d37af12b81"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.572484 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.585336 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.587704 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.587827 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.588007 8891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:53.588326 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:53.588382 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.588428 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:53.588450 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.594236 8891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42685
I20260929 18:58:53.594280 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42685 every 8 connection(s)
I20260929 18:58:53.594592 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:53.599459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 8891
I20260929 18:58:53.599576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:58:53.599789 392 external_mini_cluster.cc:1428] Setting key 3e176c5b88cb3d9890532dafe2efafb3
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:58:53.604553 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.595354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46415 (local address 127.0.98.62:37625)
0929 18:58:53.595428 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.595430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.595645 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:58:53.596606 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.596607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.596630 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.596687 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.597350 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.597851 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.598834 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.599023 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.599076 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.601427 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:53.601428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:53.601428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:53.601434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:53.602917 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.603416 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.603417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.603417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.603484 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:53.603832 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:53.603833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:53.603833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:53.603949 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:53.604008 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:53.604410 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 18:58:53.604487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:58:53.604676 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.595358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:46415)
0929 18:58:53.595520 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.595555 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.596502 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.596792 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.596795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.597045 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.597264 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.597269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.597934 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.597935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.598759 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.598761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.599094 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.599094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.599204 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.599819 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.599827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.601333 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:58:53.603010 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.603017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.603018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.603256 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.603575 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:58:53.603576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:58:53.603576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:58:53.603702 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:58:53.604031 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:58:53.604036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:58:53.604224 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 18:58:53.604318 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:58:53.604440 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260929 18:58:53.605261 9020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260929 18:58:53.605384 9020 heartbeater.cc:464] Registering TS with master...
I20260929 18:58:53.605548 9020 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
I20260929 18:58:53.605937 8561 ts_manager.cc:194] Registered new tserver with Master: 81b7604485744e059054ddfd7319580a (127.0.98.3:42685)
I20260929 18:58:53.606427 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46415
I20260929 18:58:53.611670 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:58:53.685913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8623
I20260929 18:58:53.690464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8757
I20260929 18:58:53.694466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8891
I20260929 18:58:53.698552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 8529
I20260929 18:58:53.703559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--webserver_interface=127.0.98.62
--webserver_port=46295
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:58:53.783962 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.784158 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.784179 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.785312 9029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:58:53.785355 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.785369 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.785382 9029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:58:53.785393 9029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:58:53.786921 9029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.787175 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.787437 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.792191 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.792239 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.792305 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.792621 9029 server_base.cc:1034] running on GCE node
I20260929 18:58:53.792855 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.793084 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.794219 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790708333794194 us; error 48 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.796883 9029 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:58:53.797349 9029 webserver.cc:460] Webserver started at http://127.0.98.62:46295/ using document root <none> and password file <none>
I20260929 18:58:53.797603 9029 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.797659 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.799002 9029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:53.799722 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.799898 9029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.799985 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "80a590b06ef14017879f65e2d1dbb772"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "c2b8f28b6092f41778db7c7b0e247e81"
server_key_iv: "7c48f0598ac20bbbd5aa48d290ec028c"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.800335 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.818325 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.820384 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.820534 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.824851 9029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37625
I20260929 18:58:53.824903 9096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37625 every 8 connection(s)
I20260929 18:58:53.825439 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:58:53.826678 9097 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:58:53.827359 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Bootstrap starting.
I20260929 18:58:53.827917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9029
I20260929 18:58:53.828192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42599
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37625
--webserver_port=46353
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:58:53.835623 9097 log.cc:824] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Log is configured to *not* fsync() on all Append() calls
I20260929 18:58:53.837306 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:58:53.837604 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Bootstrap complete.
I20260929 18:58:53.840020 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.840245 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a590b06ef14017879f65e2d1dbb772, State: Initialized, Role: FOLLOWER
I20260929 18:58:53.840401 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.840476 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:58:53.840524 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:58:53.840572 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:58:53.841643 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.841733 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a590b06ef14017879f65e2d1dbb772; no voters:
I20260929 18:58:53.841936 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:58:53.842005 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:58:53.842154 9101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [term 2 LEADER]: Becoming Leader. State: Replica: 80a590b06ef14017879f65e2d1dbb772, State: Running, Role: LEADER
I20260929 18:58:53.842218 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:58:53.842357 9101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260929 18:58:53.842837 9102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80a590b06ef14017879f65e2d1dbb772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260929 18:58:53.842909 9102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:53.843180 9108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:58:53.843272 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80a590b06ef14017879f65e2d1dbb772. Latest consensus state: current_term: 2 leader_uuid: "80a590b06ef14017879f65e2d1dbb772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a590b06ef14017879f65e2d1dbb772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260929 18:58:53.843364 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772 [sys.catalog]: This master's current role is: LEADER
I20260929 18:58:53.843847 9108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:58:53.844043 9108 catalog_manager.cc:1259] Loaded cluster ID: 3ae1d6c95d514137a943399ae3df44f4
I20260929 18:58:53.844221 9108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:58:53.845057 9108 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:58:53.845250 9108 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80a590b06ef14017879f65e2d1dbb772: Loaded TSK: 0
I20260929 18:58:53.845466 9108 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:58:53.919829 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:53.920058 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:53.920099 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:53.921267 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:53.921334 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:53.921380 9099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:58:53.922879 9099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46353
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:53.923128 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:53.923410 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:53.928536 9126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.928563 9123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:53.928828 9124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:53.929229 9099 server_base.cc:1034] running on GCE node
I20260929 18:58:53.929473 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:53.929724 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:53.930897 9099 hybrid_clock.cc:648] HybridClock initialized: now 1790708333930880 us; error 27 us; skew 500 ppm
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:53.933395 9099 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:58:53.933871 9099 webserver.cc:460] Webserver started at http://127.0.98.1:46353/ using document root <none> and password file <none>
I20260929 18:58:53.934100 9099 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:53.934154 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:53.935550 9099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:53.936244 9133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.936416 9099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:53.936496 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "57da0d85214c45bb8adddacbe9d8a12b"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "29ce22cf8f6f910765078509f8e03886"
server_key_iv: "1cb5294baf585c5caa7875d3765e2fa9"
server_key_version: "encryptionkey@0"
I20260929 18:58:53.936852 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:53.947867 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:53.950094 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:53.950238 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:53.950489 9099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:53.950811 9099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:53.950850 9099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.950872 9099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:53.950905 9099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:53.958081 9099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42599
I20260929 18:58:53.958159 9246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42599 every 8 connection(s)
I20260929 18:58:53.958504 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:58:53.963160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9099
I20260929 18:58:53.963488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43473
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37625
--webserver_port=40195
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:53.967533 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.959273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:39073)
0929 18:58:53.959486 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.959535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.960155 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.960603 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.960607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.961188 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.961397 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.961401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.962572 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.962573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.963467 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.963469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.964075 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.964076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.964210 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.964806 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.964818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.967317 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
W20260929 18:58:53.967592 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:53.959347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39073 (local address 127.0.98.62:37625)
0929 18:58:53.959550 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.959559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.959631 (+ 72us) server_negotiation.cc:407] Connection header received
0929 18:58:53.960380 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.960390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.960433 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.960566 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.961526 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.962456 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.963557 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.963971 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.964106 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.967436 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39073: BlockingRecv error: recv got EOF from 127.0.98.1:39073 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
W20260929 18:58:53.971479 9247 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37625 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37625: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:53.976466 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.971823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:59549)
0929 18:58:53.971907 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.971919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.972002 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.972192 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.972193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.972387 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.972448 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.972451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.973143 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.973144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.973661 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.973663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.974036 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.974036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.974109 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.974577 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.974581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.976378 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 18:58:53.976521 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:53.971858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59549 (local address 127.0.98.62:37625)
0929 18:58:53.971908 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.971911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.971983 (+ 72us) server_negotiation.cc:407] Connection header received
0929 18:58:53.972094 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.972095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.972106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.972176 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.972538 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.973056 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.973752 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.973947 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.974035 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.976457 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59549: BlockingRecv error: recv got EOF from 127.0.98.1:59549 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:53 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:53.981242 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:53.976989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:54719)
0929 18:58:53.977067 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:53.977079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:53.977127 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:53.977312 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:53.977313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:53.977525 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:53.977565 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.977567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.978190 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.978191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.978687 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:53.978689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.979062 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.979063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:53.979115 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.979578 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:53.979583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:53.981160 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:58:53.981297 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:53.977032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54719 (local address 127.0.98.62:37625)
0929 18:58:53.977070 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:53.977071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:53.977111 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:58:53.977184 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:53.977185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:53.977195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:53.977286 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:53.977649 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.978111 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.978793 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:53.978978 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:53.979060 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:53.981225 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54719: BlockingRecv error: recv got EOF from 127.0.98.1:54719 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:58:53.981606 9247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:58:54.051462 9251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:54.051676 9251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:54.051699 9251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:54.053399 9251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:54.053448 9251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:54.053475 9251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:58:54.054874 9251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40195
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:54.055052 9251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:54.055214 9251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:54.060257 9259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:54.060267 9257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:54.060241 9256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:54.060489 9251 server_base.cc:1034] running on GCE node
I20260929 18:58:54.060689 9251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:54.060892 9251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:54.062033 9251 hybrid_clock.cc:648] HybridClock initialized: now 1790708334062014 us; error 38 us; skew 500 ppm
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708334, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:54.064608 9251 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:58:54.065056 9251 webserver.cc:460] Webserver started at http://127.0.98.2:40195/ using document root <none> and password file <none>
I20260929 18:58:54.065289 9251 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:54.065346 9251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:54.066701 9251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:54.067389 9266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.067540 9251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:54.067623 9251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "e96e9a90e01c4a84abe614f0bfa9a890"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "b8fe064a6b37fbaaffafbcf2e0bce3f7"
server_key_iv: "fcfe1fe50d97fd996e01c89fac0f0aca"
server_key_version: "encryptionkey@0"
I20260929 18:58:54.067931 9251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:54.095183 9251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:54.097285 9251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:54.097394 9251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:54.097570 9251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:54.097911 9251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:54.097950 9251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.097972 9251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:54.098022 9251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.103924 9251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43473
I20260929 18:58:54.104020 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43473 every 8 connection(s)
I20260929 18:58:54.104336 9251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:58:54.109344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9251
I20260929 18:58:54.109638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42685
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37625
--webserver_port=44575
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.111514 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.105055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:38791)
0929 18:58:54.105236 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.105272 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.105713 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.105963 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.105966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.106522 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.106704 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.106708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.107498 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.107498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.108287 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.108289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.108623 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.108624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.108717 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.109132 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.109140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.111378 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260929 18:58:54.111560 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.105051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38791 (local address 127.0.98.62:37625)
0929 18:58:54.105100 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.105103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.105366 (+ 263us) server_negotiation.cc:407] Connection header received
0929 18:58:54.105828 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.105829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.105842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.105908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.106807 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.107420 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.108370 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.108549 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.108605 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.111488 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38791: BlockingRecv error: recv got EOF from 127.0.98.2:38791 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 18:58:54.111773 9380 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37625 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37625: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.116341 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.112025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:37109)
0929 18:58:54.112068 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.112081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.112108 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.112236 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.112237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.112336 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.112377 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.112378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.113265 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.113266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.113734 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.113735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.114246 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.114247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.114290 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.114709 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.114713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.116297 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 18:58:54.116473 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.112080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37109 (local address 127.0.98.62:37625)
0929 18:58:54.112118 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.112120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.112128 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:54.112151 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.112151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.112161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.112229 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.112447 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.113191 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.113819 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.114160 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.114229 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.116420 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37109: BlockingRecv error: recv got EOF from 127.0.98.2:37109 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.121042 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.116575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:60165)
0929 18:58:54.116678 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.116688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.116725 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.116872 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.116872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.116970 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.116993 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.116994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.117901 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.117902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.118358 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.118359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.118975 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.118976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.119011 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.119450 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.119454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.120978 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 18:58:54.121090 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.116678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60165 (local address 127.0.98.62:37625)
0929 18:58:54.116738 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.116740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.116777 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:58:54.116798 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.116799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.116809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.116871 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.117060 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.117817 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.118436 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.118791 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.118957 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.121051 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60165: BlockingRecv error: recv got EOF from 127.0.98.2:60165 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:58:54.121313 9380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:58:54.193073 9383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:58:54.193300 9383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:58:54.193338 9383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:58:54.194473 9383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:58:54.194536 9383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:58:54.194581 9383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:58:54.196102 9383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44575
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:58:54.196452 9383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:58:54.196664 9383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:58:54.201727 9391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:54.201721 9389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:58:54.201749 9388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:58:54.202009 9383 server_base.cc:1034] running on GCE node
I20260929 18:58:54.202310 9383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:58:54.202586 9383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:58:54.203750 9383 hybrid_clock.cc:648] HybridClock initialized: now 1790708334203738 us; error 44 us; skew 500 ppm
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708334, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:58:54.206343 9383 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:58:54.206787 9383 webserver.cc:460] Webserver started at http://127.0.98.3:44575/ using document root <none> and password file <none>
I20260929 18:58:54.207022 9383 fs_manager.cc:359] Metadata directory not provided
I20260929 18:58:54.207101 9383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:58:54.208518 9383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:54.209154 9398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.209386 9383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:58:54.209481 9383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "81b7604485744e059054ddfd7319580a"
format_stamp: "Formatted at 2026-09-29 18:58:53 on dist-test-slave-gfsd"
server_key: "143d4671a2e117b2ba790785c8c58599"
server_key_iv: "b05906db813e8aad548003d37af12b81"
server_key_version: "encryptionkey@0"
I20260929 18:58:54.209854 9383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:58:54.240654 9383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:58:54.242866 9383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:58:54.243037 9383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:58:54.243335 9383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:58:54.243649 9383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:58:54.243709 9383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.243752 9383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:58:54.243778 9383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:58:54.249588 9383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42685
I20260929 18:58:54.249637 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42685 every 8 connection(s)
I20260929 18:58:54.250005 9383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:58:54.255185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9383
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:58:54.257491 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.250777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34541 (local address 127.0.98.62:37625)
0929 18:58:54.250837 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.250839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.251048 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:58:54.251642 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.251643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.251655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.251703 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.252625 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.253134 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.254118 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.254329 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.254407 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.257447 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34541: BlockingRecv error: recv got EOF from 127.0.98.3:34541 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:58:54.257571 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.250749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:34541)
0929 18:58:54.250925 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.250962 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.251534 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.251772 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.251776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.252332 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.252518 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.252523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.253236 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.253237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.254025 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.254027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.254412 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.254413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.254527 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.254998 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.255006 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.257377 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
W20260929 18:58:54.257807 9512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37625 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37625: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.262602 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.258012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:54737)
0929 18:58:54.258048 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.258057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.258087 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.258195 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.258196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.258335 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.258384 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.258386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.259086 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.259087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.259673 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.259675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.260142 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.260143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.260209 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.260572 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.260576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.262524 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 18:58:54.262653 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.258062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54737 (local address 127.0.98.62:37625)
0929 18:58:54.258097 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.258099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.258106 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:58:54.258126 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.258127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.258135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.258190 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.258446 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.259013 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.259751 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.260045 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.260123 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.262590 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54737: BlockingRecv error: recv got EOF from 127.0.98.3:54737 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 18:58:54.263505 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.255733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.98.62:37625)
0929 18:58:54.255864 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.255866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.255874 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:58:54.255894 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.255895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.255912 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.255984 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.256781 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.257298 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.258386 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.258686 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.258763 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.260163 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:58:54.260167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:58:54.260167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:58:54.260179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:58:54.261848 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:54.262241 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:54.262244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:54.262245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:54.262373 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:58:54.262573 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:58:54.262574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:58:54.262575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:58:54.262970 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:58:54.263128 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:58:54.263253 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:58:54.263403 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.266990 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.262862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:43805)
0929 18:58:54.262935 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.262945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.262989 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.263116 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.263117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.263231 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.263261 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.263263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.264041 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.264042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.264707 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.264710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.265016 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.265017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.265051 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.265431 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.265436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.266950 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 18:58:54.267107 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.262900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43805 (local address 127.0.98.62:37625)
0929 18:58:54.262934 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.262935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.262976 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:58:54.263051 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.263051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.263060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.263113 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.263366 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.263963 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.264778 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.264953 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.265003 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.267037 (+ 2034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43805: BlockingRecv error: recv got EOF from 127.0.98.3:43805 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:58:54.267128 9512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:54 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:54.987457 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:54.982005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:42859)
0929 18:58:54.982230 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:54.982242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:54.982308 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:54.982445 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:54.982445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:54.982605 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:54.982655 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.982656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.983478 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.983479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.984039 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:54.984041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.984411 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.984412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:54.984454 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.985007 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:54.985012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:54.987338 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
W20260929 18:58:54.987535 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:54.982116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42859 (local address 127.0.98.62:37625)
0929 18:58:54.982239 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:54.982245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:54.982307 (+ 62us) server_negotiation.cc:407] Connection header received
0929 18:58:54.982352 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:54.982353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:54.982364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:54.982423 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:54.982754 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.983355 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.984135 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:54.984336 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:54.984395 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:54.987439 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42859: BlockingRecv error: recv got EOF from 127.0.98.1:42859 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:55.126711 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:55.121815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:41211)
0929 18:58:55.121953 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:55.121966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:55.122043 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:55.122275 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:55.122276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:55.122495 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:55.122544 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.122546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.123218 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.123219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.123768 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.123770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.124158 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.124159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.124222 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.124703 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:55.124708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:55.126642 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260929 18:58:55.126780 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:55.121911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41211 (local address 127.0.98.62:37625)
0929 18:58:55.121961 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:55.121963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:55.122051 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:58:55.122109 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:55.122110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:55.122123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:55.122241 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:55.122615 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.123105 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.123841 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.124050 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.124103 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.126733 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41211: BlockingRecv error: recv got EOF from 127.0.98.2:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:55.272630 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:55.267536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:59893)
0929 18:58:55.267717 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:55.267727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:55.267794 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:55.267928 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:55.267929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:55.268090 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:55.268139 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.268141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.268864 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.268865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.269431 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.269433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.269835 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.269836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.269877 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.270500 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:55.270505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:55.272514 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260929 18:58:55.272672 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:55.267653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59893 (local address 127.0.98.62:37625)
0929 18:58:55.267711 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:55.267713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:55.267774 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:58:55.267840 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:55.267840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:55.267853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:55.267913 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:55.268214 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.268783 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.269523 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.269760 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.269824 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.272589 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59893: BlockingRecv error: recv got EOF from 127.0.98.3:59893 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:55 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:55.992713 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:55.988066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:56347)
0929 18:58:55.988236 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:55.988248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:55.988301 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:55.988429 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:55.988429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:55.988587 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:55.988636 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.988638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.989276 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.989277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.989827 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:55.989835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.990190 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.990191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:55.990238 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.990792 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:55.990797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:55.992648 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20260929 18:58:55.992871 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:55.988171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56347 (local address 127.0.98.62:37625)
0929 18:58:55.988305 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:55.988307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:55.988317 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:55.988340 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:55.988341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:55.988350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:55.988411 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:55.988702 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.989195 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.989908 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:55.990101 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:55.990154 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:55.992772 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56347: BlockingRecv error: recv got EOF from 127.0.98.1:56347 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:56.135999 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:56.127265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:47609)
0929 18:58:56.127447 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:56.127458 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:56.127523 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:56.127687 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:56.127687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:56.127847 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:56.127896 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.127898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.128518 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.128519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.129066 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.129067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.129422 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.129423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.129463 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.133839 (+ 4376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:56.133844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:56.135933 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
W20260929 18:58:56.136092 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:56.127425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47609 (local address 127.0.98.62:37625)
0929 18:58:56.127469 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:56.127471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:56.127508 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:58:56.127577 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:56.127578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:56.127589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:56.127628 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:56.127965 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.128442 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.129145 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.129352 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.129410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.136027 (+ 6617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47609: BlockingRecv error: recv got EOF from 127.0.98.2:47609 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:56.278410 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:56.273127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:58915)
0929 18:58:56.273287 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:56.273306 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:56.273385 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:56.273544 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:56.273545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:56.273755 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:56.273803 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.273804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.274509 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.274511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.275063 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.275064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.275494 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.275495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.275557 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.276067 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:56.276072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:56.278290 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
W20260929 18:58:56.278452 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:56.273294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58915 (local address 127.0.98.62:37625)
0929 18:58:56.273356 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:56.273358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:56.273367 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:56.273449 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:56.273450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:56.273465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:56.273536 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:56.273884 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.274422 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.275167 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.275410 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.275492 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.278389 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58915: BlockingRecv error: recv got EOF from 127.0.98.3:58915 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:56 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:56.998585 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:56.993196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:42675)
0929 18:58:56.993329 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:56.993342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:56.993418 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:56.993592 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:56.993593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:56.993811 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:56.993873 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.993876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.995009 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.995010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.995820 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:56.995822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.996213 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.996214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:56.996267 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.996790 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:56.996794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:56.998529 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
W20260929 18:58:56.998730 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:56.993238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42675 (local address 127.0.98.62:37625)
0929 18:58:56.993333 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:56.993335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:56.993404 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:58:56.993482 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:56.993482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:56.993491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:56.993574 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:56.993970 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.994929 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.995913 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:56.996124 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:56.996213 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:56.998634 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42675: BlockingRecv error: recv got EOF from 127.0.98.1:42675 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:57.141276 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:57.136466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:53885)
0929 18:58:57.136619 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:57.136631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:57.136738 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:57.136880 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:57.136881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:57.137051 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:57.137094 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:57.137096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.137793 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.137794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:57.138372 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:57.138374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.138750 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.138751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:57.138813 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:57.139378 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:57.139385 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:57.141213 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
W20260929 18:58:57.141471 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:57.136551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53885 (local address 127.0.98.62:37625)
0929 18:58:57.136622 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:57.136625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:57.136718 (+ 93us) server_negotiation.cc:407] Connection header received
0929 18:58:57.136812 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:57.136813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:57.136825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:57.136876 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:57.137179 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.137717 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.138460 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.138677 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.138752 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:57.141398 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53885: BlockingRecv error: recv got EOF from 127.0.98.2:53885 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:57.283777 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:57.278958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:43375)
0929 18:58:57.279123 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:57.279143 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:57.279197 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:57.279410 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:57.279411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:57.279581 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:57.279634 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:57.279636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.280335 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.280336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:57.280866 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:57.280868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.281191 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.281192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:57.281240 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:57.281781 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:57.281785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:57.283713 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260929 18:58:57.283950 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:57.279055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43375 (local address 127.0.98.62:37625)
0929 18:58:57.279268 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:57.279270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:57.279294 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:58:57.279315 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:57.279316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:57.279329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:57.279385 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:57.279721 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.280252 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.280946 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:57.281120 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:57.281173 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:57.283865 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43375: BlockingRecv error: recv got EOF from 127.0.98.3:43375 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:57 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:58.004159 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:57.999073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:46707)
0929 18:58:57.999234 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:57.999246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:57.999358 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:57.999487 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:57.999487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:57.999642 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:57.999697 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:57.999698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.000380 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.000381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.000919 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:58.000921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.001357 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.001358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.001408 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.001914 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:58.001919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:58.004101 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260929 18:58:58.004278 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:57.999223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46707 (local address 127.0.98.62:37625)
0929 18:58:57.999365 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:57.999367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:57.999381 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:58:57.999403 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:57.999403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:57.999413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:57.999471 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:57.999761 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.000300 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.001017 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.001259 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.001330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.004216 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46707: BlockingRecv error: recv got EOF from 127.0.98.1:46707 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
Sep 29 18:58:58 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:58 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:58.151109 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:58.141759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:42379)
0929 18:58:58.141983 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:58.141996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:58.142070 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:58.142250 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:58.142250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:58.142498 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:58.142562 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:58.142565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.143524 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.143526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.144192 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:58.144195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.144640 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.144641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.144710 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.149033 (+ 4323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:58.149039 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:58.151046 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":120,"threads_started":1}
W20260929 18:58:58.151335 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:58.141876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42379 (local address 127.0.98.62:37625)
0929 18:58:58.141972 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:58.141975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:58.142048 (+ 73us) server_negotiation.cc:407] Connection header received
0929 18:58:58.142129 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:58.142130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:58.142143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:58.142239 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:58.142716 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.143433 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.144311 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.144521 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.144617 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.151142 (+ 6525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42379: BlockingRecv error: recv got EOF from 127.0.98.2:42379 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 18:58:58 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:58 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:58.289196 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:58.284349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:45421)
0929 18:58:58.284516 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:58.284527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:58.284585 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:58.284712 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:58.284712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:58.284883 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:58.284931 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:58.284932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.285583 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.285583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.286109 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:58.286110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.286501 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.286502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:58.286554 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.287159 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:58.287165 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:58.289134 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
W20260929 18:58:58.289356 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:58.284410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45421 (local address 127.0.98.62:37625)
0929 18:58:58.284521 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:58.284523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:58.284582 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:58:58.284628 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:58.284629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:58.284642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:58.284699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:58.285003 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.285520 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.286198 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:58.286400 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:58.286460 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:58.289248 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45421: BlockingRecv error: recv got EOF from 127.0.98.3:45421 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:59.009399 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:59.004773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:41107)
0929 18:58:59.004887 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:59.004898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:59.004952 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:59.005157 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:59.005158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:59.005320 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:59.005369 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.005371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.006057 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.006058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.006581 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.006583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.006934 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.006935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.006974 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.007510 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:59.007515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:59.009332 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260929 18:58:59.009542 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:59.004852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41107 (local address 127.0.98.62:37625)
0929 18:58:59.005019 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:59.005021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:59.005031 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:58:59.005064 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:59.005065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:59.005076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:59.005147 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:59.005477 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.005987 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.006653 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.006865 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.006920 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.009439 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41107: BlockingRecv error: recv got EOF from 127.0.98.1:41107 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":120,"threads_started":1}
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:59.157064 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:59.151749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:42889)
0929 18:58:59.151915 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:59.151934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:59.151984 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:59.152141 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:59.152142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:59.152313 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:59.152362 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.152363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.153117 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.153119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.153777 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.153781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.154240 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.154241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.154291 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.154919 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:59.154924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:59.156990 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
W20260929 18:58:59.157155 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:59.151927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42889 (local address 127.0.98.62:37625)
0929 18:58:59.152003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:59.152005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:59.152014 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:58:59.152043 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:59.152044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:59.152055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:59.152123 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:59.152469 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.153013 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.153878 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.154086 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.154150 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.157106 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42889: BlockingRecv error: recv got EOF from 127.0.98.2:42889 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:58:59 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:58:59.294649 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:58:59.289808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:47431)
0929 18:58:59.289961 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:58:59.289972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:58:59.290042 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:58:59.290174 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:58:59.290174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:58:59.290343 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:58:59.290391 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.290393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.291009 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.291009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.291666 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:58:59.291669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.292182 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.292183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:58:59.292247 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.292806 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:58:59.292811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:58:59.294582 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260929 18:58:59.294781 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:58:59.289900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47431 (local address 127.0.98.62:37625)
0929 18:58:59.289974 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:58:59.289976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:58:59.290026 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:58:59.290098 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:58:59.290099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:58:59.290111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:58:59.290152 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:58:59.290460 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.290941 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.291768 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:58:59.292085 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:58:59.292158 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:58:59.294687 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47431: BlockingRecv error: recv got EOF from 127.0.98.3:47431 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:00.015403 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:00.010028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:56803)
0929 18:59:00.010215 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:00.010231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:00.010300 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:00.010475 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:00.010476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:00.010857 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:00.010907 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.010910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.011684 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.011685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.012386 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.012388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.012756 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.012757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.012801 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.013364 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:00.013369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:00.015308 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
W20260929 18:59:00.015450 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:00.010111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56803 (local address 127.0.98.62:37625)
0929 18:59:00.010346 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:00.010347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:00.010356 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:00.010380 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:00.010380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:00.010391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:00.010459 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:00.010999 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.011609 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.012458 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.012647 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.012744 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.015380 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56803: BlockingRecv error: recv got EOF from 127.0.98.1:56803 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:00.162364 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:00.157693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:47847)
0929 18:59:00.157835 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:00.157851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:00.157906 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:00.158038 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:00.158038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:00.158195 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:00.158285 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.158287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.158960 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.158961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.159585 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.159587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.159932 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.159933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.159973 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.160462 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:00.160467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:00.162281 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260929 18:59:00.162426 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:00.157800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47847 (local address 127.0.98.62:37625)
0929 18:59:00.157845 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:00.157847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:00.157897 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:59:00.157964 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:00.157964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:00.157974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:00.158018 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:00.158371 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.158860 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.159671 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.159858 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.159916 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.162380 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47847: BlockingRecv error: recv got EOF from 127.0.98.2:47847 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:00 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:00.300351 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:00.295356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:38431)
0929 18:59:00.295573 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:00.295586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:00.295653 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:00.295850 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:00.295851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:00.296009 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:00.296058 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.296060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.296758 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.296759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.297299 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:00.297301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.297657 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.297658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:00.297718 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.298341 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:00.298346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:00.300292 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260929 18:59:00.300433 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:00.295407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38431 (local address 127.0.98.62:37625)
0929 18:59:00.295493 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:00.295496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:00.295638 (+ 142us) server_negotiation.cc:407] Connection header received
0929 18:59:00.295701 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:00.295702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:00.295716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:00.295774 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:00.296136 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.296651 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.297393 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:00.297581 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:00.297647 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:00.300385 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38431: BlockingRecv error: recv got EOF from 127.0.98.3:38431 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:01.021063 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:01.016084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:45053)
0929 18:59:01.016264 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:01.016279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:01.016350 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:01.016578 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:01.016579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:01.016757 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:01.016811 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.016813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.017470 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.017471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.017998 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.018000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.018412 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.018413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.018461 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.018928 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:01.018932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:01.020969 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20260929 18:59:01.021211 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:01.016275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45053 (local address 127.0.98.62:37625)
0929 18:59:01.016412 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:01.016414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:01.016422 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:59:01.016445 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:01.016445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:01.016454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:01.016518 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:01.016896 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.017385 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.018096 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.018311 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.018378 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.021108 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45053: BlockingRecv error: recv got EOF from 127.0.98.1:45053 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:01.167752 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:01.162917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:57295)
0929 18:59:01.163077 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:01.163089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:01.163163 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:01.163309 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:01.163310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:01.163469 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:01.163518 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.163520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.164280 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.164281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.164825 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.164827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.165202 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.165203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.165262 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.165709 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:01.165713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:01.167643 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":52,"threads_started":1}
W20260929 18:59:01.167793 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:01.163039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57295 (local address 127.0.98.62:37625)
0929 18:59:01.163088 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:01.163090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:01.163147 (+ 57us) server_negotiation.cc:407] Connection header received
0929 18:59:01.163219 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:01.163219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:01.163229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:01.163269 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:01.163617 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.164194 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.164920 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.165124 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.165206 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.167731 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57295: BlockingRecv error: recv got EOF from 127.0.98.2:57295 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:01 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:01.305876 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:01.300928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:51419)
0929 18:59:01.301111 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:01.301122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:01.301193 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:01.301354 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:01.301355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:01.301519 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:01.301567 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.301569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.302235 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.302236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.302785 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:01.302787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.303182 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.303184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:01.303245 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.303744 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:01.303748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:01.305819 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
W20260929 18:59:01.305985 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:01.300990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51419 (local address 127.0.98.62:37625)
0929 18:59:01.301062 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:01.301064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:01.301190 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:59:01.301258 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:01.301259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:01.301274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:01.301326 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:01.301634 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.302137 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.302869 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:01.303073 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:01.303128 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:01.305916 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51419: BlockingRecv error: recv got EOF from 127.0.98.3:51419 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:02.027216 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:02.021611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:41021)
0929 18:59:02.021844 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:02.021857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:02.021934 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:02.022070 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:02.022070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:02.022226 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:02.022276 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.022278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.023595 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.023597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.024150 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.024151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.024541 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.024542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.024582 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.025064 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:02.025069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:02.027120 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260929 18:59:02.027364 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:02.021799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41021 (local address 127.0.98.62:37625)
0929 18:59:02.021934 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:02.021936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:02.021946 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:59:02.021973 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:02.021974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:02.021983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:02.022045 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:02.022447 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.023479 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.024272 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.024467 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.024526 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.027199 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41021: BlockingRecv error: recv got EOF from 127.0.98.1:41021 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:02.174226 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:02.168442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:47893)
0929 18:59:02.168604 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:02.168637 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:02.168867 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:02.169058 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:02.169059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:02.169332 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:02.169387 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.169390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.170538 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.170539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.171122 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.171125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.171612 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.171613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.171673 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.172173 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:02.172178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:02.174122 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
W20260929 18:59:02.174271 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:02.168597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47893 (local address 127.0.98.62:37625)
0929 18:59:02.168667 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:02.168669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:02.168836 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:59:02.168952 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:02.168953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:02.168967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:02.169052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:02.169493 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.170450 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.171247 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.171511 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.171613 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.174207 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47893: BlockingRecv error: recv got EOF from 127.0.98.2:47893 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:02 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:02.311337 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:02.306409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:45321)
0929 18:59:02.306539 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:02.306551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:02.306617 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:02.306750 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:02.306750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:02.306908 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:02.306953 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.306955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.307665 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.307666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.308172 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:02.308174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.308676 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.308677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:02.308723 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.309277 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:02.309282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:02.311218 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
W20260929 18:59:02.311497 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:02.306514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45321 (local address 127.0.98.62:37625)
0929 18:59:02.306562 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:02.306564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:02.306603 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:59:02.306674 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:02.306675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:02.306687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:02.306737 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:02.307025 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.307593 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.308355 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:02.308582 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:02.308652 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:02.311414 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45321: BlockingRecv error: recv got EOF from 127.0.98.3:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:03.032446 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:03.027703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:53305)
0929 18:59:03.027904 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:03.027917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:03.027976 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:03.028108 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:03.028108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:03.028314 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:03.028363 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.028364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.029027 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.029028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.029602 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.029604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.029977 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.029978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.030021 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.030548 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:03.030553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:03.032389 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
W20260929 18:59:03.032651 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:03.027855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53305 (local address 127.0.98.62:37625)
0929 18:59:03.027977 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:03.027979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:03.027988 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:03.028015 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:03.028015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:03.028025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:03.028088 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:03.028436 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.028950 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.029690 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.029905 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.029963 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.032571 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53305: BlockingRecv error: recv got EOF from 127.0.98.1:53305 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:03.185778 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:03.174817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:58061)
0929 18:59:03.175025 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:03.175037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:03.175108 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:03.175308 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:03.175308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:03.175471 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:03.175518 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.175520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.176193 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.176194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.176723 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.176724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.177180 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.177181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.177237 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.183693 (+ 6456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:03.183698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:03.185685 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
W20260929 18:59:03.185850 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:03.174965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58061 (local address 127.0.98.62:37625)
0929 18:59:03.175026 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:03.175027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:03.175112 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:59:03.175172 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:03.175173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:03.175183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:03.175228 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:03.175589 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.176098 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.176804 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.176987 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.177039 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.185785 (+ 8746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58061: BlockingRecv error: recv got EOF from 127.0.98.2:58061 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:03 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:03.316764 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:03.311885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:60889)
0929 18:59:03.312053 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:03.312070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:03.312158 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:03.312304 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:03.312304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:03.312461 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:03.312510 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.312512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.313315 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.313316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.313919 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:03.313922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.314311 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.314312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:03.314356 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.314925 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:03.314930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:03.316698 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
W20260929 18:59:03.316915 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:03.311981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60889 (local address 127.0.98.62:37625)
0929 18:59:03.312047 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:03.312049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:03.312159 (+ 110us) server_negotiation.cc:407] Connection header received
0929 18:59:03.312205 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:03.312206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:03.312221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:03.312297 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:03.312581 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.313195 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.314016 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:03.314222 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:03.314297 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:03.316833 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60889: BlockingRecv error: recv got EOF from 127.0.98.3:60889 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:04.037815 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:04.032970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:33189)
0929 18:59:04.033164 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:04.033176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:04.033247 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:04.033407 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:04.033407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:04.033571 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:04.033619 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.033620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.034281 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.034283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.034843 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.034845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.035240 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.035241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.035348 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.035870 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:04.035875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:04.037736 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
W20260929 18:59:04.037922 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:04.033062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33189 (local address 127.0.98.62:37625)
0929 18:59:04.033164 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:04.033166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:04.033247 (+ 81us) server_negotiation.cc:407] Connection header received
0929 18:59:04.033309 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:04.033309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:04.033320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:04.033382 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:04.033686 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.034191 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.034927 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.035136 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.035191 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.037845 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33189: BlockingRecv error: recv got EOF from 127.0.98.1:33189 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:04.191172 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:04.186358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:41119)
0929 18:59:04.186522 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:04.186536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:04.186605 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:04.186742 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:04.186743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:04.186924 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:04.186975 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.186976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.187654 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.187655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.188229 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.188231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.188567 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.188568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.188610 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.189114 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:04.189119 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:04.191109 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":57,"threads_started":1}
W20260929 18:59:04.191416 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:04.186538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41119 (local address 127.0.98.62:37625)
0929 18:59:04.186616 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:04.186618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:04.186626 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:59:04.186651 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:04.186652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:04.186661 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:04.186713 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:04.187050 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.187580 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.188318 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.188497 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.188546 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.191230 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41119: BlockingRecv error: recv got EOF from 127.0.98.2:41119 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:04 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:04.321852 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:04.317230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:44583)
0929 18:59:04.317404 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:04.317417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:04.317483 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:04.317652 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:04.317653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:04.317820 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:04.317867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.317869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.318513 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.318514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.319039 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:04.319041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.319374 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.319375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:04.319423 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.319976 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:04.319980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:04.321745 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
W20260929 18:59:04.321946 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:04.317383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44583 (local address 127.0.98.62:37625)
0929 18:59:04.317447 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:04.317449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:04.317468 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:59:04.317539 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:04.317540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:04.317550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:04.317601 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:04.317943 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.318433 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.319113 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:04.319304 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:04.319351 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:04.321877 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44583: BlockingRecv error: recv got EOF from 127.0.98.3:44583 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:05.044045 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:05.038363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:51431)
0929 18:59:05.038528 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:05.038541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:05.038602 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:05.038763 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:05.038764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:05.038940 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:05.039028 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.039031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.039891 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.039892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.040639 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.040658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.041169 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.041170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.041230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.041744 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:05.041748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:05.043955 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
W20260929 18:59:05.044131 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:05.038474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51431 (local address 127.0.98.62:37625)
0929 18:59:05.038601 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:05.038603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:05.038612 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:05.038657 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:05.038658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:05.038667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:05.038723 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:05.039114 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.039794 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.040829 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.041040 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.041169 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.044058 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51431: BlockingRecv error: recv got EOF from 127.0.98.1:51431 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":52,"threads_started":1}
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:05.197129 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:05.191821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:56397)
0929 18:59:05.192149 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:05.192164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:05.192254 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:05.192457 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:05.192458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:05.192651 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:05.192709 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.192710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.193499 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.193501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.194218 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.194221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.194630 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.194631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.194689 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.195188 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:05.195194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:05.197071 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":117,"threads_started":1}
W20260929 18:59:05.197288 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:05.191946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56397 (local address 127.0.98.62:37625)
0929 18:59:05.192133 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:05.192136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:05.192325 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:59:05.192362 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:05.192363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:05.192376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:05.192437 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:05.192804 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.193413 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.194313 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.194533 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.194631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.197206 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56397: BlockingRecv error: recv got EOF from 127.0.98.2:56397 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:05 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:05.327487 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:05.322374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:47343)
0929 18:59:05.322529 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:05.322541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:05.322609 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:05.322744 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:05.322744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:05.322903 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:05.322952 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.322953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.323644 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.323645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.324280 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:05.324283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.324791 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.324792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:05.324851 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.325395 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:05.325400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:05.327425 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260929 18:59:05.327561 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:05.322435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47343 (local address 127.0.98.62:37625)
0929 18:59:05.322517 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:05.322519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:05.322614 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:59:05.322663 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:05.322664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:05.322676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:05.322719 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:05.323026 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.323542 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.324420 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:05.324691 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:05.324781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:05.327514 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47343: BlockingRecv error: recv got EOF from 127.0.98.3:47343 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:06.049510 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:06.044602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:42313)
0929 18:59:06.044746 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:06.044757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:06.044809 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:06.044951 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:06.044952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:06.045179 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:06.045257 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.045259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.046202 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.046203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.046734 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.046735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.047076 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.047077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.047126 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.047629 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:06.047634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:06.049452 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260929 18:59:06.049628 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:06.044674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42313 (local address 127.0.98.62:37625)
0929 18:59:06.044829 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:06.044831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:06.044839 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:59:06.044861 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:06.044862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:06.044871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:06.044935 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:06.045402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.046115 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.046809 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.046999 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.047052 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.049547 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42313: BlockingRecv error: recv got EOF from 127.0.98.1:42313 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:06.202919 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:06.197671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:59159)
0929 18:59:06.197816 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:06.197829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:06.197908 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:06.198080 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:06.198081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:06.198299 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:06.198352 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.198354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.199058 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.199059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.199675 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.199677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.200087 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.200088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.200135 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.200700 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:06.200706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:06.202852 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
W20260929 18:59:06.203045 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:06.197803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59159 (local address 127.0.98.62:37625)
0929 18:59:06.197874 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:06.197876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:06.197886 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:59:06.197987 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:06.197988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:06.198001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:06.198057 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:06.198455 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.198961 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.199770 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.199963 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.200026 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.202945 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59159: BlockingRecv error: recv got EOF from 127.0.98.2:59159 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:06 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:06.335654 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:06.328017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:49297)
0929 18:59:06.328165 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:06.328177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:06.328245 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:06.328507 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:06.328508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:06.328726 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:06.328788 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.328790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.330300 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.330302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.331449 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:06.331456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.332033 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.332034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:06.332101 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.332806 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:06.332819 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:06.335553 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260929 18:59:06.335779 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:06.328137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49297 (local address 127.0.98.62:37625)
0929 18:59:06.328213 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:06.328215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:06.328228 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:59:06.328385 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:06.328386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:06.328402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:06.328481 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:06.328854 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.330172 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.331545 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:06.331941 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:06.332020 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:06.335694 (+ 3674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49297: BlockingRecv error: recv got EOF from 127.0.98.3:49297 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:07.054643 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:07.050079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:57547)
0929 18:59:07.050196 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:07.050212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:07.050262 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:07.050411 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:07.050412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:07.050558 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:07.050606 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.050608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.051267 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.051268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.051837 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.051840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.052188 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.052189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.052234 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.052682 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:07.052687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:07.054576 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
W20260929 18:59:07.054795 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:07.050138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57547 (local address 127.0.98.62:37625)
0929 18:59:07.050269 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:07.050271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:07.050280 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:07.050307 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:07.050308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:07.050320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:07.050390 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:07.050673 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.051194 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.051913 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.052109 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.052166 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.054672 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57547: BlockingRecv error: recv got EOF from 127.0.98.1:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:07.208719 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:07.203453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:33139)
0929 18:59:07.203612 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:07.203623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:07.203710 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:07.203877 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:07.203878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:07.204094 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:07.204170 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.204172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.204863 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.204864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.205419 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.205421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.205819 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.205819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.205874 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.206395 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:07.206400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:07.208621 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
W20260929 18:59:07.208765 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:07.203615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33139 (local address 127.0.98.62:37625)
0929 18:59:07.203680 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:07.203682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:07.203690 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:59:07.203808 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:07.203809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:07.203821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:07.203874 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:07.204262 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.204762 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.205527 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.205731 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.205819 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.208701 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33139: BlockingRecv error: recv got EOF from 127.0.98.2:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:07 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:07.342918 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:07.337284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:60489)
0929 18:59:07.337435 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:07.337452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:07.337512 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:07.337653 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:07.337653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:07.337828 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:07.337875 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.337877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.338675 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.338677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.339654 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:07.339658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.340134 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.340135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:07.340178 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.340895 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:07.340900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:07.342843 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260929 18:59:07.342970 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:07.337404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60489 (local address 127.0.98.62:37625)
0929 18:59:07.337468 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:07.337470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:07.337498 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:59:07.337572 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:07.337573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:07.337585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:07.337628 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:07.337976 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.338475 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.339756 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:07.340051 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:07.340119 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:07.342919 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60489: BlockingRecv error: recv got EOF from 127.0.98.3:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:08.060238 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:08.055143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:32915)
0929 18:59:08.055308 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:08.055321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:08.055383 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:08.055664 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:08.055665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:08.055846 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:08.055893 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.055895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.056599 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.056599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.057150 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.057152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.057527 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.057528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.057575 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.058177 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:08.058183 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:08.060175 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260929 18:59:08.060436 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:08.055363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32915 (local address 127.0.98.62:37625)
0929 18:59:08.055500 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:08.055502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:08.055511 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:08.055537 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:08.055538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:08.055549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:08.055624 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:08.055969 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.056508 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.057246 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.057445 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.057509 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.060365 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32915: BlockingRecv error: recv got EOF from 127.0.98.1:32915 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:08.216996 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:08.209231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:33489)
0929 18:59:08.209437 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:08.209451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:08.209525 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:08.209668 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:08.209669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:08.209858 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:08.209905 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.209907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.210560 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.210561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.211105 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.211106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.211500 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.211501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.211563 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.214879 (+ 3316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:08.214884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:08.216933 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":159,"threads_started":1}
W20260929 18:59:08.217196 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:08.209342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33489 (local address 127.0.98.62:37625)
0929 18:59:08.209414 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:08.209416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:08.209525 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:59:08.209575 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:08.209576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:08.209587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:08.209646 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:08.209971 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.210477 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.211181 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.211403 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.211462 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.217116 (+ 5654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33489: BlockingRecv error: recv got EOF from 127.0.98.2:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:08 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:08.348279 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:08.343407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:57797)
0929 18:59:08.343553 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:08.343564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:08.343632 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:08.343804 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:08.343805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:08.343962 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:08.344010 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.344012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.344678 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.344679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.345198 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:08.345199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.345549 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.345550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:08.345589 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.346051 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:08.346056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:08.348213 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260929 18:59:08.348467 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:08.343484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57797 (local address 127.0.98.62:37625)
0929 18:59:08.343529 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:08.343531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:08.343612 (+ 81us) server_negotiation.cc:407] Connection header received
0929 18:59:08.343714 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:08.343715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:08.343730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:08.343785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:08.344073 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.344603 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.345267 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:08.345480 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:08.345534 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:08.348340 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57797: BlockingRecv error: recv got EOF from 127.0.98.3:57797 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 18:59:09 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:09 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:59:09.065848 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.060832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:51067)
0929 18:59:09.061010 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:09.061021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:09.061074 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:09.061215 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:09.061216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:09.061401 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:09.061454 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.061456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.062176 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.062177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.062765 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.062768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.063333 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.063335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.063404 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.063875 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:09.063880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:09.065779 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
W20260929 18:59:09.065949 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:09.060946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51067 (local address 127.0.98.62:37625)
0929 18:59:09.061064 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.061067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.061087 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:59:09.061118 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.061119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.061129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.061193 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.061556 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.062086 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.062861 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.063218 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.063320 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.065875 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51067: BlockingRecv error: recv got EOF from 127.0.98.1:51067 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
Sep 29 18:59:09 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:59:09 dist-test-slave-gfsd krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:59:09.222793 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:09.217591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52419 (local address 127.0.98.62:37625)
0929 18:59:09.217637 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.217639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.217771 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:59:09.217860 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.217866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.217878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.217938 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.218346 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.218950 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.219858 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.220066 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.220210 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.222741 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52419: BlockingRecv error: recv got EOF from 127.0.98.2:52419 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:59:09.222810 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.217534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:52419)
0929 18:59:09.217689 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:09.217700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:09.217790 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:09.217945 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:09.217947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:09.218126 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:09.218174 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.218176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.219071 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.219072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.219767 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.219769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.220183 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.220185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.220282 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.220744 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:09.220749 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:09.222657 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37625: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260929 18:59:09.259223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9099
I20260929 18:59:09.263978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9251
I20260929 18:59:09.268401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9383
I20260929 18:59:09.272547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9029
2026-09-29T18:59:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17459 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:59:09Z Disabled control of system clock
I20260929 18:59:09.299914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:59:09.379871 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:09.380074 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:09.380095 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:09.381268 9579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:59:09.381314 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:09.381327 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:09.381340 9579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:59:09.381352 9579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:59:09.382686 9579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:09.382949 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:09.383247 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:09.388091 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.388063 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.388239 9579 server_base.cc:1034] running on GCE node
W20260929 18:59:09.388180 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.388586 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:09.388896 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:09.390077 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790708349390065 us; error 53 us; skew 500 ppm
I20260929 18:59:09.391536 9579 webserver.cc:460] Webserver started at http://127.0.98.62:41539/ using document root <none> and password file <none>
I20260929 18:59:09.391803 9579 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:09.391892 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:09.392059 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:09.393111 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "a7525a8570644f858bb7ecd78bb39903"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf08a3d9b6f5bc9f75e13b867701fee3"
server_key_iv: "b9bfb54f7880f48f49fbec534044e855"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.393560 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7525a8570644f858bb7ecd78bb39903"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf08a3d9b6f5bc9f75e13b867701fee3"
server_key_iv: "b9bfb54f7880f48f49fbec534044e855"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.394846 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:59:09.395612 9593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.395792 9579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:09.395897 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "a7525a8570644f858bb7ecd78bb39903"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf08a3d9b6f5bc9f75e13b867701fee3"
server_key_iv: "b9bfb54f7880f48f49fbec534044e855"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.395987 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:09.405004 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:09.405292 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:09.405435 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:09.409375 9579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36299
I20260929 18:59:09.409409 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36299 every 8 connection(s)
I20260929 18:59:09.409739 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:59:09.410459 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:09.413056 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903: Bootstrap starting.
I20260929 18:59:09.413636 9646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:09.413921 9646 log.cc:824] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:09.413949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9579
I20260929 18:59:09.414047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:59:09.414297 392 external_mini_cluster.cc:1428] Setting key 952289f39cdf96b55fcb11ac5d2bd4c9
I20260929 18:59:09.414613 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903: No bootstrap required, opened a new log
I20260929 18:59:09.416399 9646 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260929 18:59:09.416524 9646 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:09.416544 9646 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7525a8570644f858bb7ecd78bb39903, State: Initialized, Role: FOLLOWER
I20260929 18:59:09.416687 9646 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260929 18:59:09.416791 9646 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:59:09.416834 9646 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:59:09.416903 9646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:09.417619 9646 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260929 18:59:09.417725 9646 leader_election.cc:302] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7525a8570644f858bb7ecd78bb39903; no voters:
I20260929 18:59:09.417868 9646 leader_election.cc:288] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:59:09.417941 9651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:09.418079 9651 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [term 1 LEADER]: Becoming Leader. State: Replica: a7525a8570644f858bb7ecd78bb39903, State: Running, Role: LEADER
I20260929 18:59:09.418092 9646 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:59:09.418224 9651 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } }
I20260929 18:59:09.418561 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7525a8570644f858bb7ecd78bb39903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } }
I20260929 18:59:09.418632 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:09.418857 9658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:59:09.419585 9658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:59:09.419631 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7525a8570644f858bb7ecd78bb39903. Latest consensus state: current_term: 1 leader_uuid: "a7525a8570644f858bb7ecd78bb39903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7525a8570644f858bb7ecd78bb39903" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } }
I20260929 18:59:09.419790 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903 [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:09.421214 9658 catalog_manager.cc:1347] Generated new cluster ID: c08d71cf4f064a96abcb6a6a53e27166
I20260929 18:59:09.421289 9658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:59:09.421947 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.414801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.0.98.62:36299)
0929 18:59:09.415035 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.415043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.415081 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:59:09.416418 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.416426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.416488 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.416661 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.416906 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.418654 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.419923 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.420397 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.420550 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.421238 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:09.421241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:09.421241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:09.421455 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:09.421487 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:09.421498 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:09.421575 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:09.421653 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:59:09.421729 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260929 18:59:09.434167 9658 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:59:09.434666 9658 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:59:09.437595 9658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7525a8570644f858bb7ecd78bb39903: Generated new TSK 0
I20260929 18:59:09.437769 9658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:59:09.440694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--builtin_ntp_servers=127.0.98.20:35113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:59:09.529600 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:09.529820 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:09.529857 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:09.531003 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:09.531064 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:09.531105 9670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:59:09.532516 9670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:09.532771 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:09.533067 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:09.538879 9675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.538923 9676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.538887 9678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.539197 9670 server_base.cc:1034] running on GCE node
I20260929 18:59:09.539475 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:09.539716 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:09.540869 9670 hybrid_clock.cc:648] HybridClock initialized: now 1790708349540845 us; error 39 us; skew 500 ppm
I20260929 18:59:09.542119 9670 webserver.cc:460] Webserver started at http://127.0.98.1:36321/ using document root <none> and password file <none>
I20260929 18:59:09.542358 9670 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:09.542471 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:09.542632 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:09.543763 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca9a281e691744d182c0d633fc2761f4"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf90791589a9fa179f3f1d360e2a1f90"
server_key_iv: "c8d75ee95997d1d1a84a25162f0d2a07"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.544131 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca9a281e691744d182c0d633fc2761f4"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf90791589a9fa179f3f1d360e2a1f90"
server_key_iv: "c8d75ee95997d1d1a84a25162f0d2a07"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.545528 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 18:59:09.546355 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.546533 9670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:09.546629 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "ca9a281e691744d182c0d633fc2761f4"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "bf90791589a9fa179f3f1d360e2a1f90"
server_key_iv: "c8d75ee95997d1d1a84a25162f0d2a07"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.546715 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:09.572762 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:09.573113 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:09.573297 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:09.573598 9670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:09.573963 9670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:09.574019 9670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.574062 9670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:09.574091 9670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.579918 9670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43933
I20260929 18:59:09.579964 9797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43933 every 8 connection(s)
I20260929 18:59:09.580327 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:59:09.585645 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.581111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47037 (local address 127.0.98.62:36299)
0929 18:59:09.581159 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.581161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.581414 (+ 253us) server_negotiation.cc:407] Connection header received
0929 18:59:09.581974 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.581976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.581990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.582051 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.582710 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.583218 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.584226 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.584428 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.584512 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.585325 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:09.585326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:09.585326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:09.585332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:09.585349 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:09.585359 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:09.585427 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:09.585534 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:59:09.585564 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:59:09.585873 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.581071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.1:47037)
0929 18:59:09.581249 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:09.581283 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:09.581854 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:09.582119 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:09.582124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:09.582134 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:09.582614 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.582618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.583330 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.583331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.584122 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.584123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.584521 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.584522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.584657 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.585183 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:09.585184 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:09.585212 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:09.585234 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:09.585442 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:09.585447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:09.585450 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:59:09.585538 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:59:09.585637 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260929 18:59:09.586434 9798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260929 18:59:09.586524 9798 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:09.586565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9670
I20260929 18:59:09.586683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:59:09.586736 9798 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260929 18:59:09.586907 392 external_mini_cluster.cc:1428] Setting key 95ba533fa383d03db515371c240035ba
I20260929 18:59:09.587356 9610 ts_manager.cc:194] Registered new tserver with Master: ca9a281e691744d182c0d633fc2761f4 (127.0.98.1:43933)
I20260929 18:59:09.588021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--builtin_ntp_servers=127.0.98.20:35113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:59:09.588079 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47037
W20260929 18:59:09.667774 9801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:09.667979 9801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:09.667999 9801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:09.669148 9801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:09.669195 9801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:09.669224 9801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:59:09.670557 9801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:09.670802 9801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:09.671105 9801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:09.676266 9809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.676425 9801 server_base.cc:1034] running on GCE node
W20260929 18:59:09.676491 9807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.676451 9806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.676890 9801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:09.677085 9801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:09.678227 9801 hybrid_clock.cc:648] HybridClock initialized: now 1790708349678214 us; error 30 us; skew 500 ppm
I20260929 18:59:09.679484 9801 webserver.cc:460] Webserver started at http://127.0.98.2:35963/ using document root <none> and password file <none>
I20260929 18:59:09.679713 9801 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:09.679759 9801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:09.679934 9801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:09.680949 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "f34ff512fbc648e1907845b6dd7f17fa"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "7dae8a5b073134668c2583c590f6dbf8"
server_key_iv: "cbca210fc2595267a1f0709d9432b9ee"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.681623 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f34ff512fbc648e1907845b6dd7f17fa"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "7dae8a5b073134668c2583c590f6dbf8"
server_key_iv: "cbca210fc2595267a1f0709d9432b9ee"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.683022 9801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:59:09.683849 9815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.684091 9801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:59:09.684161 9801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "f34ff512fbc648e1907845b6dd7f17fa"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "7dae8a5b073134668c2583c590f6dbf8"
server_key_iv: "cbca210fc2595267a1f0709d9432b9ee"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.684253 9801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:09.696923 9801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:09.697222 9801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:09.697368 9801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:09.697602 9801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:09.697923 9801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:09.697985 9801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.698027 9801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:09.698055 9801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.704223 9801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39235
I20260929 18:59:09.704283 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39235 every 8 connection(s)
I20260929 18:59:09.704643 9801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:59:09.709842 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.705366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49511 (local address 127.0.98.62:36299)
0929 18:59:09.705414 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.705416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.705628 (+ 212us) server_negotiation.cc:407] Connection header received
0929 18:59:09.706169 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.706170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.706180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.706256 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.706967 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.707533 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.708512 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.708697 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.708757 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.709611 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:09.709612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:09.709612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:09.709616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:09.709629 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:09.709633 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:09.709679 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:09.709762 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:59:09.709785 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:59:09.710080 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.705332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.2:49511)
0929 18:59:09.705515 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:09.705553 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:09.706052 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:09.706357 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:09.706361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:09.706369 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:09.706864 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.706868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.707615 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.707616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.708425 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.708428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.708777 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.708777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.708901 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.709466 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:09.709466 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:09.709494 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:09.709516 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:09.709692 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:09.709696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:09.709698 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:59:09.709761 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:59:09.709861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260929 18:59:09.710724 9929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260929 18:59:09.710807 9929 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:09.710985 9929 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260929 18:59:09.711376 9610 ts_manager.cc:194] Registered new tserver with Master: f34ff512fbc648e1907845b6dd7f17fa (127.0.98.2:39235)
I20260929 18:59:09.711757 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49511
I20260929 18:59:09.712747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9801
I20260929 18:59:09.712833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:59:09.713054 392 external_mini_cluster.cc:1428] Setting key 5784a0712d1b1e4ca60fa9efbadcf1d2
I20260929 18:59:09.714006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--builtin_ntp_servers=127.0.98.20:35113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:59:09.795610 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:09.795804 9932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:09.795825 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:09.796922 9932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:09.796957 9932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:09.796981 9932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:59:09.798287 9932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:09.798691 9932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:09.799088 9932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:09.804097 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.804118 9932 server_base.cc:1034] running on GCE node
W20260929 18:59:09.804121 9937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.804103 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.804541 9932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:09.804831 9932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:09.806008 9932 hybrid_clock.cc:648] HybridClock initialized: now 1790708349806003 us; error 39 us; skew 500 ppm
I20260929 18:59:09.807199 9932 webserver.cc:460] Webserver started at http://127.0.98.3:41813/ using document root <none> and password file <none>
I20260929 18:59:09.807541 9932 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:09.807615 9932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:09.807765 9932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:09.808867 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac253271e6bd429ab0f31341cc01fc4b"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "3813b5377011032c891c0da53e5dd912"
server_key_iv: "4357d29b394f835a63eaa984e31d6c55"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.809213 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac253271e6bd429ab0f31341cc01fc4b"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "3813b5377011032c891c0da53e5dd912"
server_key_iv: "4357d29b394f835a63eaa984e31d6c55"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.810519 9932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:59:09.811528 9946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.811713 9932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:09.811812 9932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "ac253271e6bd429ab0f31341cc01fc4b"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "3813b5377011032c891c0da53e5dd912"
server_key_iv: "4357d29b394f835a63eaa984e31d6c55"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.811892 9932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:09.834138 9932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:09.834462 9932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:09.834619 9932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:09.834853 9932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:09.835244 9932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:09.835338 9932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.835402 9932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:09.835433 9932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.841439 9932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33489
I20260929 18:59:09.841476 10059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33489 every 8 connection(s)
I20260929 18:59:09.841809 9932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:59:09.847141 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.842563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46863 (local address 127.0.98.62:36299)
0929 18:59:09.842627 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.842629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.842864 (+ 235us) server_negotiation.cc:407] Connection header received
0929 18:59:09.843425 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.843426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.843437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.843500 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.844255 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.844739 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.845769 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.845961 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.846023 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.846901 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:09.846903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:09.846903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:09.846907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:09.846921 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:09.846924 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:09.846973 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:09.847053 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:59:09.847080 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:59:09.847398 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.842512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.98.3:46863)
0929 18:59:09.842710 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:09.842751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:09.843237 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:09.843619 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:09.843623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:09.843631 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:09.844137 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.844141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.844859 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.844861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.845680 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:09.845682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:09.846043 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:09.846043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:09.846151 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:09.846757 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:09.846758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:09.846786 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:09.846810 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:09.846987 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:09.846990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:09.846992 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:59:09.847044 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:59:09.847124 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260929 18:59:09.847965 10060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36299
I20260929 18:59:09.848070 10060 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:09.848248 10060 heartbeater.cc:511] Master 127.0.98.62:36299 requested a full tablet report, sending...
I20260929 18:59:09.848658 9610 ts_manager.cc:194] Registered new tserver with Master: ac253271e6bd429ab0f31341cc01fc4b (127.0.98.3:33489)
I20260929 18:59:09.848881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 9932
I20260929 18:59:09.848995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:59:09.849133 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46863
I20260929 18:59:09.849198 392 external_mini_cluster.cc:1428] Setting key 12399f1d5a3b2906a336278f1477f338
I20260929 18:59:09.850100 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:59:09.851878 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:09.851388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.0.98.62:36299)
0929 18:59:09.851449 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.851451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.851604 (+ 153us) server_negotiation.cc:407] Connection header received
0929 18:59:09.851625 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.851626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.851636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.851684 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.851814 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42284: BlockingRecv error: recv got EOF from 127.0.0.1:42284 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:59:09.851943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9670
I20260929 18:59:09.857182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9801
I20260929 18:59:09.861246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9932
I20260929 18:59:09.865446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 9579
2026-09-29T18:59:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (591 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:59:09Z Disabled control of system clock
I20260929 18:59:09.891260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39333
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:59:09.970933 10071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:09.971127 10071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:09.971148 10071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:09.972301 10071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:59:09.972342 10071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:09.972355 10071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:09.972368 10071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:59:09.972380 10071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:59:09.973639 10071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:09.973816 10071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:09.974038 10071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:09.979087 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.979116 10071 server_base.cc:1034] running on GCE node
W20260929 18:59:09.979087 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:09.979142 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:09.979642 10071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:09.979941 10071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:09.981120 10071 hybrid_clock.cc:648] HybridClock initialized: now 1790708349981114 us; error 35 us; skew 500 ppm
I20260929 18:59:09.982445 10071 webserver.cc:460] Webserver started at http://127.0.98.62:42873/ using document root <none> and password file <none>
I20260929 18:59:09.982697 10071 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:09.982769 10071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:09.982926 10071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:09.984047 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "eaa81380e9f34561a2259e12e90b451f"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "616b4d1752de7b13e143da122e24adf1"
server_key_iv: "3bd0dd203fbede738a2bde9ee409a461"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.984408 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "eaa81380e9f34561a2259e12e90b451f"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "616b4d1752de7b13e143da122e24adf1"
server_key_iv: "3bd0dd203fbede738a2bde9ee409a461"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.985793 10071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:59:09.986881 10085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:09.987051 10071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:09.987142 10071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "eaa81380e9f34561a2259e12e90b451f"
format_stamp: "Formatted at 2026-09-29 18:59:09 on dist-test-slave-gfsd"
server_key: "616b4d1752de7b13e143da122e24adf1"
server_key_iv: "3bd0dd203fbede738a2bde9ee409a461"
server_key_version: "encryptionkey@0"
I20260929 18:59:09.987195 10071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:09.990463 10071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:09.990669 10071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:09.990761 10071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:09.994289 10071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39333
I20260929 18:59:09.994329 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39333 every 8 connection(s)
I20260929 18:59:09.994637 10071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:59:09.995059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10071
I20260929 18:59:09.995157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:59:09.995422 10138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:09.995456 392 external_mini_cluster.cc:1428] Setting key 4b41673d78f45139cb69f038040e87db
I20260929 18:59:09.998186 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:09.996154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.0.98.62:39333)
0929 18:59:09.996357 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:09.996366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:09.996397 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:59:09.996901 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:09.996910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:09.996965 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:09.997111 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:09.997772 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:09.997775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:09.997776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:09.997780 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:09.997826 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:09.997839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:09.997920 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:09.997955 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 18:59:09.998008 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260929 18:59:09.998672 10138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: Bootstrap starting.
I20260929 18:59:09.999341 10138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:09.999588 10138 log.cc:824] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:10.000228 10138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: No bootstrap required, opened a new log
I20260929 18:59:10.001726 10138 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } }
I20260929 18:59:10.001876 10138 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:10.001900 10138 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa81380e9f34561a2259e12e90b451f, State: Initialized, Role: FOLLOWER
I20260929 18:59:10.001981 10138 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } }
I20260929 18:59:10.002039 10138 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:59:10.002060 10138 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:59:10.002086 10138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:10.002562 10138 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } }
I20260929 18:59:10.002657 10138 leader_election.cc:302] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa81380e9f34561a2259e12e90b451f; no voters:
I20260929 18:59:10.002769 10138 leader_election.cc:288] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:59:10.003010 10138 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:59:10.003026 10143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:10.003154 10143 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [term 1 LEADER]: Becoming Leader. State: Replica: eaa81380e9f34561a2259e12e90b451f, State: Running, Role: LEADER
I20260929 18:59:10.003333 10143 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } }
I20260929 18:59:10.003769 10145 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa81380e9f34561a2259e12e90b451f. Latest consensus state: current_term: 1 leader_uuid: "eaa81380e9f34561a2259e12e90b451f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } } }
I20260929 18:59:10.003867 10145 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:10.004197 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa81380e9f34561a2259e12e90b451f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa81380e9f34561a2259e12e90b451f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39333 } } }
I20260929 18:59:10.004271 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f [sys.catalog]: This master's current role is: LEADER
W20260929 18:59:10.004825 10158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:59:10.005153 10158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:59:10.005213 10152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:59:10.005384 10152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:59:10.006713 10152 catalog_manager.cc:1347] Generated new cluster ID: ee09556e5f564e8a927bdee60686afb3
I20260929 18:59:10.006763 10152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:59:10.020701 10152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:59:10.020889 10152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:59:10.025195 10152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa81380e9f34561a2259e12e90b451f: Generated new TSK 0
I20260929 18:59:10.025384 10152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:59:10.029767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:59:10.111465 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:10.111682 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:10.111707 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:10.112886 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:10.112931 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:10.112959 10162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:59:10.114323 10162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:10.114566 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:10.114889 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:10.120826 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:10.120800 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.120996 10162 server_base.cc:1034] running on GCE node
W20260929 18:59:10.120805 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.121418 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:10.121690 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:10.122888 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790708350122860 us; error 43 us; skew 500 ppm
I20260929 18:59:10.124378 10162 webserver.cc:460] Webserver started at http://127.0.98.1:36527/ using document root <none> and password file <none>
I20260929 18:59:10.124614 10162 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:10.124681 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:10.124848 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:10.125910 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "c04fa82fc6814bf49fa0aa0da4671eac"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "177869d39cfd5b6c37b03e21715e1df7"
server_key_iv: "544af39ce6e8a78904b6fa8414028d4e"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.126271 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c04fa82fc6814bf49fa0aa0da4671eac"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "177869d39cfd5b6c37b03e21715e1df7"
server_key_iv: "544af39ce6e8a78904b6fa8414028d4e"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.127683 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:59:10.128582 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.128791 10162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:59:10.128918 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "c04fa82fc6814bf49fa0aa0da4671eac"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "177869d39cfd5b6c37b03e21715e1df7"
server_key_iv: "544af39ce6e8a78904b6fa8414028d4e"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.129009 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:10.132855 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:10.133140 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:10.133306 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:10.133540 10162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:10.133872 10162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:10.133931 10162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.133973 10162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:10.134003 10162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.140239 10162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43849
I20260929 18:59:10.140317 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43849 every 8 connection(s)
I20260929 18:59:10.140671 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:59:10.143653 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.141427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40099 (local address 127.0.98.62:39333)
0929 18:59:10.141505 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:10.141508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:10.141717 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:59:10.142154 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:10.142155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:10.142160 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:10.142223 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:10.143419 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:10.143420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:10.143420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:10.143425 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:10.143439 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:10.143443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:10.143497 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:10.143588 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:59:10.143600 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:59:10.143818 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.141396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39333 (local address 127.0.98.1:40099)
0929 18:59:10.141598 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:10.141631 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:10.142068 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:10.142276 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:10.142279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:10.142286 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:10.143164 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:10.143164 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:10.143191 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:10.143217 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:10.143513 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:10.143517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:10.143520 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:59:10.143583 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:59:10.143631 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260929 18:59:10.144325 10290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39333
I20260929 18:59:10.144447 10290 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:10.144593 10290 heartbeater.cc:511] Master 127.0.98.62:39333 requested a full tablet report, sending...
I20260929 18:59:10.144578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10162
I20260929 18:59:10.144651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:59:10.144873 392 external_mini_cluster.cc:1428] Setting key 3d5243f9b6d771461d9a140b5b7437dd
I20260929 18:59:10.145135 10102 ts_manager.cc:194] Registered new tserver with Master: c04fa82fc6814bf49fa0aa0da4671eac (127.0.98.1:43849)
I20260929 18:59:10.145967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:59:10.227783 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:10.227989 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:10.228013 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:10.229092 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:10.229132 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:10.229159 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:59:10.230525 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:10.230715 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:10.230967 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:10.236183 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:10.236225 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.236407 10293 server_base.cc:1034] running on GCE node
W20260929 18:59:10.236306 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.236686 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:10.236956 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:10.238106 10293 hybrid_clock.cc:648] HybridClock initialized: now 1790708350238094 us; error 33 us; skew 500 ppm
I20260929 18:59:10.239522 10293 webserver.cc:460] Webserver started at http://127.0.98.2:36693/ using document root <none> and password file <none>
I20260929 18:59:10.239756 10293 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:10.239815 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:10.239945 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:10.241076 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "c77c605d85de48cc99098cd6adc4ec4a"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "e62575c1a75790ffa7175eef3091666e"
server_key_iv: "1ccc9ae9e5e543eaf000cdffdf6e657b"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.241453 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c77c605d85de48cc99098cd6adc4ec4a"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "e62575c1a75790ffa7175eef3091666e"
server_key_iv: "1ccc9ae9e5e543eaf000cdffdf6e657b"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.242772 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:59:10.243660 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.243831 10293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.243942 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "c77c605d85de48cc99098cd6adc4ec4a"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "e62575c1a75790ffa7175eef3091666e"
server_key_iv: "1ccc9ae9e5e543eaf000cdffdf6e657b"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.244019 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:10.247476 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:10.247750 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:10.247900 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:10.248117 10293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:10.248437 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:10.248493 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.248534 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:10.248565 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.255156 10293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43269
I20260929 18:59:10.255193 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43269 every 8 connection(s)
I20260929 18:59:10.255558 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:59:10.258620 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.256286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51557 (local address 127.0.98.62:39333)
0929 18:59:10.256351 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:10.256353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:10.256582 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:59:10.257105 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:10.257107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:10.257115 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:10.257181 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:10.258317 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:10.258319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:10.258319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:10.258326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:10.258343 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:10.258348 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:10.258415 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:10.258522 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:59:10.258539 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:59:10.258822 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.256283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39333 (local address 127.0.98.2:51557)
0929 18:59:10.256459 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:10.256491 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:10.256985 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:10.257261 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:10.257265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:10.257274 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:10.258154 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:10.258154 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:10.258181 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:10.258207 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:10.258424 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:10.258429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:10.258432 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:59:10.258533 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:59:10.258594 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260929 18:59:10.259311 10421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39333
I20260929 18:59:10.259433 10421 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:10.259546 10421 heartbeater.cc:511] Master 127.0.98.62:39333 requested a full tablet report, sending...
I20260929 18:59:10.259886 10101 ts_manager.cc:194] Registered new tserver with Master: c77c605d85de48cc99098cd6adc4ec4a (127.0.98.2:43269)
I20260929 18:59:10.260526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10293
I20260929 18:59:10.260637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:59:10.260838 392 external_mini_cluster.cc:1428] Setting key cc0f5feb8d7dbad58d3d74c51abb4c44
I20260929 18:59:10.261797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:59:10.341248 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:10.341449 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:10.341472 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:10.342561 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:10.342593 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:10.342618 10424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:59:10.343901 10424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:10.344076 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:10.344298 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:10.349400 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.349458 10424 server_base.cc:1034] running on GCE node
W20260929 18:59:10.349423 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:10.349390 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:10.349793 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:10.349987 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:10.351161 10424 hybrid_clock.cc:648] HybridClock initialized: now 1790708350351146 us; error 34 us; skew 500 ppm
I20260929 18:59:10.352639 10424 webserver.cc:460] Webserver started at http://127.0.98.3:36429/ using document root <none> and password file <none>
I20260929 18:59:10.352921 10424 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:10.352974 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:10.353145 10424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:10.354279 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "a9545649cd7e4635a10335b02d625abc"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "9eaba0963e7bb07738496e690f6a1c6d"
server_key_iv: "78231961312a3a02660102481ccb104f"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.354637 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a9545649cd7e4635a10335b02d625abc"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "9eaba0963e7bb07738496e690f6a1c6d"
server_key_iv: "78231961312a3a02660102481ccb104f"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.356070 10424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:59:10.356897 10438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.357093 10424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:10.357162 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "a9545649cd7e4635a10335b02d625abc"
format_stamp: "Formatted at 2026-09-29 18:59:10 on dist-test-slave-gfsd"
server_key: "9eaba0963e7bb07738496e690f6a1c6d"
server_key_iv: "78231961312a3a02660102481ccb104f"
server_key_version: "encryptionkey@0"
I20260929 18:59:10.357317 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:10.361110 10424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:10.361387 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:10.361510 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:10.361750 10424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:10.362036 10424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:10.362067 10424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.362118 10424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:10.362145 10424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:10.367991 10424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33921
I20260929 18:59:10.368031 10551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33921 every 8 connection(s)
I20260929 18:59:10.368340 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:59:10.371361 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.369038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52569 (local address 127.0.98.62:39333)
0929 18:59:10.369103 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:10.369105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:10.369339 (+ 234us) server_negotiation.cc:407] Connection header received
0929 18:59:10.369775 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:10.369776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:10.369781 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:10.369858 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:10.371052 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:10.371054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:10.371054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:59:10.371058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:10.371075 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:59:10.371079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:10.371139 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:10.371198 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:59:10.371211 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:59:10.371563 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:10.369023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39333 (local address 127.0.98.3:52569)
0929 18:59:10.369210 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:10.369250 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:10.369689 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:10.369909 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:10.369912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:10.369919 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:10.370898 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:59:10.370898 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:10.370926 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:59:10.370952 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:10.371146 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:10.371149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:10.371151 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:59:10.371204 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:59:10.371247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260929 18:59:10.372144 10552 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39333
I20260929 18:59:10.372263 10552 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:10.372422 10552 heartbeater.cc:511] Master 127.0.98.62:39333 requested a full tablet report, sending...
I20260929 18:59:10.372789 10101 ts_manager.cc:194] Registered new tserver with Master: a9545649cd7e4635a10335b02d625abc (127.0.98.3:33921)
I20260929 18:59:10.376663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10424
I20260929 18:59:10.376775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:59:10.376973 392 external_mini_cluster.cc:1428] Setting key b4818abc14519a5d1263444325403647
I20260929 18:59:10.377777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:59:10.379700 10141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:59:10.379163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.0.98.62:39333)
0929 18:59:10.379245 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:10.379248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:10.379377 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:59:10.379457 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:10.379458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:10.379462 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:10.379511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:10.379640 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58096: BlockingRecv error: recv got EOF from 127.0.0.1:58096 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:59:10.379774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10162
I20260929 18:59:10.384553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10293
I20260929 18:59:10.388964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10424
I20260929 18:59:10.393576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10071
2026-09-29T18:59:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (528 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:59:10 dist-test-slave-gfsd krb5kdc[10562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:59:10 dist-test-slave-gfsd krb5kdc[10562](info): set up 2 sockets
Sep 29 18:59:10 dist-test-slave-gfsd krb5kdc[10562](info): commencing operation
krb5kdc: starting...
W20260929 18:59:11.428310 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:59:11.545871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:11.625566 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:11.625759 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:11.625782 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:11.626924 10575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:59:11.626964 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:11.626979 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:11.626994 10575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:59:11.627030 10575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:59:11.628459 10575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:11.628693 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:11.628921 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:11.633755 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:11.633769 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:11.633967 10575 server_base.cc:1034] running on GCE node
W20260929 18:59:11.633823 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:11.634299 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:11.634591 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:11.635756 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790708351635744 us; error 46 us; skew 500 ppm
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:11.638234 10575 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:59:11.638710 10575 webserver.cc:460] Webserver started at http://127.0.98.62:43909/ using document root <none> and password file <none>
I20260929 18:59:11.638954 10575 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:11.639025 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:11.639181 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:11.640372 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "b5d4e653b73642b7b065a9c43788bb1a"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "d903c5dc41577c7dd310d3d7178ef66b"
server_key_iv: "9dd8bd48c45e12e639cc59cb62ad22a2"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.640722 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5d4e653b73642b7b065a9c43788bb1a"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "d903c5dc41577c7dd310d3d7178ef66b"
server_key_iv: "9dd8bd48c45e12e639cc59cb62ad22a2"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.642045 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:59:11.642838 10590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.643023 10575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:59:11.643127 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "b5d4e653b73642b7b065a9c43788bb1a"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "d903c5dc41577c7dd310d3d7178ef66b"
server_key_iv: "9dd8bd48c45e12e639cc59cb62ad22a2"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.643209 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:11.652133 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:11.653874 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:11.654026 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:11.658246 10575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42487
I20260929 18:59:11.658301 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42487 every 8 connection(s)
I20260929 18:59:11.658638 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:59:11.659715 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:11.660199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10575
I20260929 18:59:11.660312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:59:11.660493 392 external_mini_cluster.cc:1428] Setting key f329eff66b7d5657f93af9fd3da4dc41
I20260929 18:59:11.662667 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: Bootstrap starting.
I20260929 18:59:11.663239 10643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:11.663563 10643 log.cc:824] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:11.664291 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: No bootstrap required, opened a new log
I20260929 18:59:11.666756 10643 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } }
I20260929 18:59:11.666896 10643 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:11.666923 10643 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d4e653b73642b7b065a9c43788bb1a, State: Initialized, Role: FOLLOWER
I20260929 18:59:11.667054 10643 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } }
I20260929 18:59:11.667145 10643 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:59:11.667168 10643 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:59:11.667236 10643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:11.667878 10643 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } }
I20260929 18:59:11.668006 10643 leader_election.cc:302] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d4e653b73642b7b065a9c43788bb1a; no voters:
I20260929 18:59:11.668162 10643 leader_election.cc:288] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:59:11.668193 10648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:11.668421 10643 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:59:11.668680 10648 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [term 1 LEADER]: Becoming Leader. State: Replica: b5d4e653b73642b7b065a9c43788bb1a, State: Running, Role: LEADER
I20260929 18:59:11.668776 10648 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } }
I20260929 18:59:11.669493 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d4e653b73642b7b065a9c43788bb1a. Latest consensus state: current_term: 1 leader_uuid: "b5d4e653b73642b7b065a9c43788bb1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } } }
I20260929 18:59:11.669574 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:11.669647 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d4e653b73642b7b065a9c43788bb1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d4e653b73642b7b065a9c43788bb1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42487 } } }
I20260929 18:59:11.669728 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:11.669761 10657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:59:11.670280 10657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:59:11.671820 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:11.661013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.98.62:42487)
0929 18:59:11.661172 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:11.661179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:11.661202 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:59:11.661584 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:11.661590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:11.661638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:11.661749 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:11.662547 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.663557 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.664583 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.664979 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.665059 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:11.667960 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:11.667965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:11.667965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:11.667977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:11.670116 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:11.670538 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:11.670541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:11.670541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:11.670639 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:11.670859 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:11.670860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:11.670860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:11.671124 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:11.671261 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:11.671453 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:59:11.671584 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260929 18:59:11.672248 10657 catalog_manager.cc:1347] Generated new cluster ID: 011d80cf8296468a8a04393894b11962
I20260929 18:59:11.672312 10657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:59:11.681579 10657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:59:11.682104 10657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:59:11.686595 10657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: Generated new TSK 0
I20260929 18:59:11.686812 10657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:59:11.718327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:11.800308 10669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:11.800554 10669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:11.800595 10669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:11.801792 10669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:11.801858 10669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:11.801903 10669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:59:11.803459 10669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:11.803725 10669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:11.803957 10669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:11.809314 10677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:11.809286 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:11.809280 10674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:11.809473 10669 server_base.cc:1034] running on GCE node
I20260929 18:59:11.809744 10669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:11.809957 10669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:11.811131 10669 hybrid_clock.cc:648] HybridClock initialized: now 1790708351811114 us; error 27 us; skew 500 ppm
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:11.813874 10669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:59:11.814359 10669 webserver.cc:460] Webserver started at http://127.0.98.1:46521/ using document root <none> and password file <none>
I20260929 18:59:11.814613 10669 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:11.814682 10669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:11.814842 10669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:11.816015 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "5625a6ee0a094b419245b9539a064263"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "a05e1997a0994410d7f521ab427c8932"
server_key_iv: "a48d2e96e0de6d29adfdee3c14019d74"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.816386 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5625a6ee0a094b419245b9539a064263"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "a05e1997a0994410d7f521ab427c8932"
server_key_iv: "a48d2e96e0de6d29adfdee3c14019d74"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.817750 10669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:59:11.818528 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.818694 10669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:59:11.818801 10669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "5625a6ee0a094b419245b9539a064263"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "a05e1997a0994410d7f521ab427c8932"
server_key_iv: "a48d2e96e0de6d29adfdee3c14019d74"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.818905 10669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:11.842574 10669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:11.845315 10669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:11.845556 10669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:11.845839 10669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:11.846172 10669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:11.846232 10669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.846277 10669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:11.846339 10669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.852522 10669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43705
I20260929 18:59:11.852563 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43705 every 8 connection(s)
I20260929 18:59:11.852918 10669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:59:11.853370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10669
I20260929 18:59:11.853478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:59:11.853714 392 external_mini_cluster.cc:1428] Setting key 8a7433bd8ab36e3afddf0b816856a318
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:59:11.864271 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:11.853823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52065 (local address 127.0.98.62:42487)
0929 18:59:11.853869 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:11.853872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:11.854089 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:59:11.854663 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:11.854664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:11.854676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:11.854732 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:11.855843 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.856662 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.858262 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.858526 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.858593 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:11.861157 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:11.861158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:11.861159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:11.861167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:11.862707 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:11.863259 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:11.863260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:11.863261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:11.863344 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:11.863610 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:11.863611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:11.863611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:11.863724 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:11.863799 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:11.864087 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 18:59:11.864179 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:59:11.864450 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:11.853800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42487 (local address 127.0.98.1:52065)
0929 18:59:11.853965 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:11.854008 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:11.854551 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:11.854811 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:11.854815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:11.855469 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:11.855704 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:11.855711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.856788 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.856790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:11.858174 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:11.858177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:11.858623 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:11.858624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:11.858778 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:11.859309 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:11.859329 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:11.861044 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:11.862930 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:11.862936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:11.862938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:11.863180 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:11.863428 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:11.863429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:11.863429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:11.863536 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:11.863791 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:11.863820 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:11.863956 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:59:11.864067 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:59:11.864225 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260929 18:59:11.865254 10798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:59:11.865386 10798 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:11.865597 10798 heartbeater.cc:511] Master 127.0.98.62:42487 requested a full tablet report, sending...
I20260929 18:59:11.866245 10607 ts_manager.cc:194] Registered new tserver with Master: 5625a6ee0a094b419245b9539a064263 (127.0.98.1:43705)
I20260929 18:59:11.867185 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:59:11.882004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:11.963518 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:11.963724 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:11.963749 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:11.964941 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:11.964990 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:11.965052 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:59:11.966501 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:11.966807 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:11.967070 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:11.972620 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:11.972815 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:11.972765 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:11.972854 10803 server_base.cc:1034] running on GCE node
I20260929 18:59:11.973155 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:11.973415 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:11.974589 10803 hybrid_clock.cc:648] HybridClock initialized: now 1790708351974574 us; error 43 us; skew 500 ppm
Sep 29 18:59:11 dist-test-slave-gfsd krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:11.977245 10803 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:59:11.977644 10803 webserver.cc:460] Webserver started at http://127.0.98.2:39437/ using document root <none> and password file <none>
I20260929 18:59:11.977862 10803 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:11.977905 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:11.977993 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:11.979153 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "7cb3923ab2554d69a4acd5ac118a836f"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "582da1d582bcc5033080401a4badf071"
server_key_iv: "05b941a74f03f218ab0e1081dacfdecd"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.979511 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7cb3923ab2554d69a4acd5ac118a836f"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "582da1d582bcc5033080401a4badf071"
server_key_iv: "05b941a74f03f218ab0e1081dacfdecd"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.980808 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:11.981600 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.981801 10803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:11.981884 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "7cb3923ab2554d69a4acd5ac118a836f"
format_stamp: "Formatted at 2026-09-29 18:59:11 on dist-test-slave-gfsd"
server_key: "582da1d582bcc5033080401a4badf071"
server_key_iv: "05b941a74f03f218ab0e1081dacfdecd"
server_key_version: "encryptionkey@0"
I20260929 18:59:11.981932 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:11.991762 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:11.993973 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:11.994094 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:11.994269 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:11.994609 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:11.994642 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:11.994670 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:11.994683 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:12.000540 10803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35883
I20260929 18:59:12.000612 10931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35883 every 8 connection(s)
I20260929 18:59:12.000918 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:59:12.006650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10803
I20260929 18:59:12.006767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:59:12.006976 392 external_mini_cluster.cc:1428] Setting key 72078bffa896ef291aaa6a306187da5b
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:12.010646 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.001674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44461 (local address 127.0.98.62:42487)
0929 18:59:12.001742 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.001745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.001955 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:59:12.002440 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.002441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.002452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.002506 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:12.003534 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.004019 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.005028 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.005206 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.005313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.007731 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:12.007732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:12.007732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:12.007738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:12.009195 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.009683 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.009684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.009684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.009732 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.010020 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.010020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.010021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.010119 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:12.010171 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.010519 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.010585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:59:12.010912 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.001654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42487 (local address 127.0.98.2:44461)
0929 18:59:12.001835 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.001871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.002348 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.002579 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.002582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.003169 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:12.003456 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.003461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.004101 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.004102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.004949 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.004951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.005334 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.005335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.005440 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.005904 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:12.005912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:12.007621 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:12.009279 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:12.009285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:12.009287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:12.009606 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:12.009798 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:12.009799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:12.009800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:12.009956 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:12.010169 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:12.010174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:12.010339 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.010464 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.010637 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260929 18:59:12.011534 10932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42487
I20260929 18:59:12.011642 10932 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:12.011835 10932 heartbeater.cc:511] Master 127.0.98.62:42487 requested a full tablet report, sending...
I20260929 18:59:12.012192 10607 ts_manager.cc:194] Registered new tserver with Master: 7cb3923ab2554d69a4acd5ac118a836f (127.0.98.2:35883)
I20260929 18:59:12.012569 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44461
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:59:12.034192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:12.114888 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:12.115095 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:12.115118 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:12.116281 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:12.116323 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:12.116349 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:59:12.117746 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:12.117976 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:12.118214 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:12.123118 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:12.123186 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:12.123169 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:12.123262 10937 server_base.cc:1034] running on GCE node
I20260929 18:59:12.123771 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:12.123970 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:12.125110 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790708352125098 us; error 29 us; skew 500 ppm
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:12.127602 10937 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:59:12.128018 10937 webserver.cc:460] Webserver started at http://127.0.98.3:33823/ using document root <none> and password file <none>
I20260929 18:59:12.128285 10937 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:12.128353 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:12.128486 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:59:12.129496 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "61aaeec1731040cfaf9d5c1ef252c1e4"
format_stamp: "Formatted at 2026-09-29 18:59:12 on dist-test-slave-gfsd"
server_key: "e2160ba0d688f43d60db0be634d91717"
server_key_iv: "38875f1d9034cfac79d1353537a9a71c"
server_key_version: "encryptionkey@0"
I20260929 18:59:12.129859 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61aaeec1731040cfaf9d5c1ef252c1e4"
format_stamp: "Formatted at 2026-09-29 18:59:12 on dist-test-slave-gfsd"
server_key: "e2160ba0d688f43d60db0be634d91717"
server_key_iv: "38875f1d9034cfac79d1353537a9a71c"
server_key_version: "encryptionkey@0"
I20260929 18:59:12.131132 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:59:12.131951 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:12.132125 10937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:12.132220 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "61aaeec1731040cfaf9d5c1ef252c1e4"
format_stamp: "Formatted at 2026-09-29 18:59:12 on dist-test-slave-gfsd"
server_key: "e2160ba0d688f43d60db0be634d91717"
server_key_iv: "38875f1d9034cfac79d1353537a9a71c"
server_key_version: "encryptionkey@0"
I20260929 18:59:12.132310 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:12.145704 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:12.147964 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:12.148129 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:12.148450 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:12.148788 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:12.148854 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:12.148896 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:12.148922 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:12.154927 10937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46221
I20260929 18:59:12.154994 11065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46221 every 8 connection(s)
I20260929 18:59:12.155373 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:59:12.158689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 10937
I20260929 18:59:12.158808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:59:12.159008 392 external_mini_cluster.cc:1428] Setting key c83c218afca2de174af121cc1ef33d3d
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:12.164754 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.156139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45645 (local address 127.0.98.62:42487)
0929 18:59:12.156180 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.156182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.156465 (+ 283us) server_negotiation.cc:407] Connection header received
0929 18:59:12.156981 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.156982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.156993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.157060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:12.157972 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.158486 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.159763 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.159985 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.160070 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.162107 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:12.162109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:12.162109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:12.162117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:12.163409 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.163810 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.163811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.163811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.163870 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.164116 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.164117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.164117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.164235 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:12.164306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.164588 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.164664 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:59:12.164949 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.156105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42487 (local address 127.0.98.3:45645)
0929 18:59:12.156281 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.156316 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.156877 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.157136 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.157139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.157682 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:12.157871 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.157875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.158599 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.158600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.159664 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.159666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.160079 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.160080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.160209 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.160634 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:12.160642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:12.162009 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:12.163501 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:12.163507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:12.163509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:12.163723 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:12.163937 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:12.163938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:12.163938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:12.164046 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:12.164307 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:12.164314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:12.164458 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.164568 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.164717 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260929 18:59:12.165359 11066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42487
I20260929 18:59:12.165446 11066 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:12.165571 11066 heartbeater.cc:511] Master 127.0.98.62:42487 requested a full tablet report, sending...
I20260929 18:59:12.165930 10607 ts_manager.cc:194] Registered new tserver with Master: 61aaeec1731040cfaf9d5c1ef252c1e4 (127.0.98.3:46221)
I20260929 18:59:12.166254 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45645
I20260929 18:59:12.170795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:59:12.178107 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.172224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.98.62:42487)
0929 18:59:12.172269 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.172271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.172356 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:59:12.172442 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.172443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.172453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.172518 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:12.173062 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.173535 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.174283 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.174485 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.174559 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.175790 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:12.175791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:12.175792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:12.175798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:12.176920 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.177293 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.177295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.177295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.177354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:12.177556 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:12.177558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:12.177558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:12.177702 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:12.177805 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.177925 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.178008 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:59:12.179630 10607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:59:12.181149 10607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:59:12.188143 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.183694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43705 (local address 127.0.0.1:51904)
0929 18:59:12.184003 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.184023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.184109 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.184402 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.184405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.184407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:12.184472 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.184473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185843 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.185844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.186874 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.186877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.187841 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.187924 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.187938 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.188006 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.188064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":88,"thread_start_us":71,"threads_started":1}
I20260929 18:59:12.188285 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.183692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46221 (local address 127.0.0.1:44526)
0929 18:59:12.183982 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.184022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.184122 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.184488 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.184489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.184491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:12.184585 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.184588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185818 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.185819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.187143 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.187146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.188036 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.188037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.188110 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.188114 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.188171 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.188214 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":51,"thread_start_us":50,"threads_started":1}
I20260929 18:59:12.188302 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.183775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.0.98.3:46221)
0929 18:59:12.183935 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.183940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.184090 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:59:12.184218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.184220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.184234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.184386 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:12.184648 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185722 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187240 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.187951 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.188036 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.188083 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.188185 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.188220 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260929 18:59:12.188485 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.183797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.98.1:43705)
0929 18:59:12.184154 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.184159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.184169 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:59:12.184211 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.184213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.184229 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.184299 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:12.184534 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185711 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.186966 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.187739 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187828 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.187885 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.188340 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.188384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":86,"threads_started":1}
I20260929 18:59:12.188869 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.183921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35883 (local address 127.0.0.1:37814)
0929 18:59:12.184394 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.184405 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.184457 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.184729 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.184729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.184731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:12.184767 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.184768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185762 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.185763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.187183 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.187184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.187956 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.188034 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.188039 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.188085 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.188117 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":39,"thread_start_us":50,"threads_started":1}
I20260929 18:59:12.189066 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.184133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.98.2:35883)
0929 18:59:12.184330 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.184334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.184515 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:59:12.184557 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.184560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.184577 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.184657 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:12.184859 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.185645 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187267 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.187767 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.187868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.187926 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.188181 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.188804 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260929 18:59:12.188993 11000 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 (DEFAULT_TABLE table=test-table [id=172e93d1b0bf41659df0c5ae1a2bd256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:12.189214 10732 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 (DEFAULT_TABLE table=test-table [id=172e93d1b0bf41659df0c5ae1a2bd256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:12.189357 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f09a395eb425087ba76cfc426b6a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:12.189510 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f09a395eb425087ba76cfc426b6a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:12.189687 10866 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 (DEFAULT_TABLE table=test-table [id=172e93d1b0bf41659df0c5ae1a2bd256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:12.189996 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f09a395eb425087ba76cfc426b6a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:12.192533 11080 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Bootstrap starting.
I20260929 18:59:12.192812 11081 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Bootstrap starting.
I20260929 18:59:12.193220 11080 tablet_bootstrap.cc:654] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:12.193425 11081 tablet_bootstrap.cc:654] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:12.193747 11080 log.cc:824] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:12.193943 11081 log.cc:824] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:12.194275 11082 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: Bootstrap starting.
I20260929 18:59:12.194474 11080 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: No bootstrap required, opened a new log
I20260929 18:59:12.194561 11080 ts_tablet_manager.cc:1397] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:59:12.194664 11081 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: No bootstrap required, opened a new log
I20260929 18:59:12.194731 11081 ts_tablet_manager.cc:1397] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:59:12.194913 11082 tablet_bootstrap.cc:654] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:12.195488 11082 log.cc:824] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:12.196233 11082 tablet_bootstrap.cc:492] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: No bootstrap required, opened a new log
I20260929 18:59:12.196310 11082 ts_tablet_manager.cc:1397] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:59:12.196619 11080 raft_consensus.cc:348] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.196777 11080 raft_consensus.cc:374] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:12.196820 11080 raft_consensus.cc:729] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb3923ab2554d69a4acd5ac118a836f, State: Initialized, Role: FOLLOWER
I20260929 18:59:12.196813 11081 raft_consensus.cc:348] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.196985 11081 raft_consensus.cc:374] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:12.196942 11080 consensus_queue.cc:260] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.197027 11081 raft_consensus.cc:729] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61aaeec1731040cfaf9d5c1ef252c1e4, State: Initialized, Role: FOLLOWER
I20260929 18:59:12.197151 11081 consensus_queue.cc:260] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.197260 11080 ts_tablet_manager.cc:1428] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:59:12.197377 10932 heartbeater.cc:503] Master 127.0.98.62:42487 was elected leader, sending a full tablet report...
I20260929 18:59:12.197403 11081 ts_tablet_manager.cc:1428] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:59:12.197510 11066 heartbeater.cc:503] Master 127.0.98.62:42487 was elected leader, sending a full tablet report...
I20260929 18:59:12.198531 11082 raft_consensus.cc:348] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.198654 11082 raft_consensus.cc:374] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:12.198676 11082 raft_consensus.cc:729] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5625a6ee0a094b419245b9539a064263, State: Initialized, Role: FOLLOWER
I20260929 18:59:12.198784 11082 consensus_queue.cc:260] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.199057 11082 ts_tablet_manager.cc:1428] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:59:12.199127 10798 heartbeater.cc:503] Master 127.0.98.62:42487 was elected leader, sending a full tablet report...
I20260929 18:59:12.202200 11086 raft_consensus.cc:482] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:59:12.202266 11086 raft_consensus.cc:504] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.202558 11086 leader_election.cc:288] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5625a6ee0a094b419245b9539a064263 (127.0.98.1:43705), 61aaeec1731040cfaf9d5c1ef252c1e4 (127.0.98.3:46221)
I20260929 18:59:12.206329 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.202750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48319 (local address 127.0.98.1:43705)
0929 18:59:12.202788 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.202790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.202826 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:59:12.202913 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.202913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.202922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.202970 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:12.203140 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.203900 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.205423 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.205987 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.206061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.206069 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.206236 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.206270 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:59:12.206336 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.202750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46999 (local address 127.0.98.3:46221)
0929 18:59:12.202788 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.202790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.203027 (+ 237us) server_negotiation.cc:407] Connection header received
0929 18:59:12.203061 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.203062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.203069 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.203138 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:12.203260 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.203962 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.205423 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.205987 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.206061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.206069 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.206237 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.206270 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:59:12.206359 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.202712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46221 (local address 127.0.98.2:46999)
0929 18:59:12.202943 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.202950 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.203014 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.203142 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.203143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.203144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:12.203194 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.203195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.204043 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.204044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.205330 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.205333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.206072 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.206073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.206153 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.206157 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.206221 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.206267 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":85,"thread_start_us":77,"threads_started":1}
I20260929 18:59:12.206416 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.202712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43705 (local address 127.0.98.2:48319)
0929 18:59:12.202766 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.202779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:12.202847 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:12.202997 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:12.202999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:12.203001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:12.203073 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.203075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.204023 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.204024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.205333 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:12.205335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.206072 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.206073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:12.206158 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.206161 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:59:12.206221 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:59:12.206267 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":55}
I20260929 18:59:12.206678 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f09a395eb425087ba76cfc426b6a3" candidate_uuid: "7cb3923ab2554d69a4acd5ac118a836f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" is_pre_election: true
I20260929 18:59:12.206674 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f09a395eb425087ba76cfc426b6a3" candidate_uuid: "7cb3923ab2554d69a4acd5ac118a836f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625a6ee0a094b419245b9539a064263" is_pre_election: true
I20260929 18:59:12.206794 11020 raft_consensus.cc:2454] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb3923ab2554d69a4acd5ac118a836f in term 0.
I20260929 18:59:12.206794 10752 raft_consensus.cc:2454] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb3923ab2554d69a4acd5ac118a836f in term 0.
I20260929 18:59:12.206977 10820 leader_election.cc:302] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61aaeec1731040cfaf9d5c1ef252c1e4, 7cb3923ab2554d69a4acd5ac118a836f; no voters:
I20260929 18:59:12.207115 11086 raft_consensus.cc:2781] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:59:12.207170 11086 raft_consensus.cc:482] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:59:12.207192 11086 raft_consensus.cc:3037] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:12.207893 11086 raft_consensus.cc:504] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.208015 11086 leader_election.cc:288] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [CANDIDATE]: Term 1 election: Requested vote from peers 5625a6ee0a094b419245b9539a064263 (127.0.98.1:43705), 61aaeec1731040cfaf9d5c1ef252c1e4 (127.0.98.3:46221)
I20260929 18:59:12.208175 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f09a395eb425087ba76cfc426b6a3" candidate_uuid: "7cb3923ab2554d69a4acd5ac118a836f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625a6ee0a094b419245b9539a064263"
I20260929 18:59:12.208214 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f09a395eb425087ba76cfc426b6a3" candidate_uuid: "7cb3923ab2554d69a4acd5ac118a836f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4"
I20260929 18:59:12.208267 10752 raft_consensus.cc:3037] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:12.208264 11020 raft_consensus.cc:3037] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:12.208885 11020 raft_consensus.cc:2454] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb3923ab2554d69a4acd5ac118a836f in term 1.
I20260929 18:59:12.208998 10752 raft_consensus.cc:2454] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb3923ab2554d69a4acd5ac118a836f in term 1.
I20260929 18:59:12.209030 10820 leader_election.cc:302] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61aaeec1731040cfaf9d5c1ef252c1e4, 7cb3923ab2554d69a4acd5ac118a836f; no voters:
I20260929 18:59:12.209115 11086 raft_consensus.cc:2781] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:12.209244 11086 raft_consensus.cc:686] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 1 LEADER]: Becoming Leader. State: Replica: 7cb3923ab2554d69a4acd5ac118a836f, State: Running, Role: LEADER
I20260929 18:59:12.209339 11086 consensus_queue.cc:237] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } }
I20260929 18:59:12.210080 10605 catalog_manager.cc:5495] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb3923ab2554d69a4acd5ac118a836f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7cb3923ab2554d69a4acd5ac118a836f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cb3923ab2554d69a4acd5ac118a836f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35883 } health_report { overall_health: HEALTHY } } }
I20260929 18:59:12.223470 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.221044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.0.98.2:35883)
0929 18:59:12.221106 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:12.221107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:12.221118 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:59:12.221175 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:12.221176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:12.221210 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:12.221288 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:59:12.221458 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.221901 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.222741 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:12.222929 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:12.222983 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:12.223109 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:59:12.223198 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:59:12.223256 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:12.223353 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:59:12.223370 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:59:12.225013 10752 raft_consensus.cc:1261] T fc2f09a395eb425087ba76cfc426b6a3 P 5625a6ee0a094b419245b9539a064263 [term 1 FOLLOWER]: Refusing update from remote peer 7cb3923ab2554d69a4acd5ac118a836f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:59:12.225117 11020 raft_consensus.cc:1261] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 1 FOLLOWER]: Refusing update from remote peer 7cb3923ab2554d69a4acd5ac118a836f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:59:12.225325 11090 consensus_queue.cc:1035] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5625a6ee0a094b419245b9539a064263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:59:12.225425 11086 consensus_queue.cc:1035] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [LEADER]: Connected to new peer: Peer: permanent_uuid: "61aaeec1731040cfaf9d5c1ef252c1e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:59:12.227721 11094 mvcc.cc:205] Tried to move back new op lower bound from 7334741410711490560 to 7334741410649948160. Current Snapshot: MvccSnapshot[applied={T|T < 7334741410711490560}]
I20260929 18:59:12.230360 10606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:59:12.230458 10606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:59:12.231208 10606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d4e653b73642b7b065a9c43788bb1a: Sending DeleteTablet for 3 replicas of tablet fc2f09a395eb425087ba76cfc426b6a3
I20260929 18:59:12.231585 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44526
I20260929 18:59:12.231597 10866 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37814
I20260929 18:59:12.231633 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f09a395eb425087ba76cfc426b6a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51904
I20260929 18:59:12.231818 11102 tablet_replica.cc:330] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: stopping tablet replica
I20260929 18:59:12.231956 11102 raft_consensus.cc:2229] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:59:12.231967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10669
I20260929 18:59:12.232051 11102 raft_consensus.cc:2258] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:59:12.235414 11103 tablet_replica.cc:330] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: stopping tablet replica
I20260929 18:59:12.235558 11103 raft_consensus.cc:2229] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:59:12.235697 11103 raft_consensus.cc:2258] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:59:12.236066 11102 ts_tablet_manager.cc:1905] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:59:12.236222 11103 ts_tablet_manager.cc:1905] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:59:12.237800 11102 ts_tablet_manager.cc:1918] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:59:12.237928 11102 log.cc:1196] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/fc2f09a395eb425087ba76cfc426b6a3
I20260929 18:59:12.238096 11103 ts_tablet_manager.cc:1918] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:59:12.238174 11103 log.cc:1196] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/fc2f09a395eb425087ba76cfc426b6a3
W20260929 18:59:12.238256 10593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43705 (error 108)
I20260929 18:59:12.238430 11102 ts_tablet_manager.cc:1939] T fc2f09a395eb425087ba76cfc426b6a3 P 61aaeec1731040cfaf9d5c1ef252c1e4: Deleting consensus metadata
I20260929 18:59:12.238504 11103 ts_tablet_manager.cc:1939] T fc2f09a395eb425087ba76cfc426b6a3 P 7cb3923ab2554d69a4acd5ac118a836f: Deleting consensus metadata
I20260929 18:59:12.238912 10592 catalog_manager.cc:4841] TS 61aaeec1731040cfaf9d5c1ef252c1e4 (127.0.98.3:46221): tablet fc2f09a395eb425087ba76cfc426b6a3 (table test-table [id=172e93d1b0bf41659df0c5ae1a2bd256]) successfully deleted
I20260929 18:59:12.238947 10593 catalog_manager.cc:4841] TS 7cb3923ab2554d69a4acd5ac118a836f (127.0.98.2:35883): tablet fc2f09a395eb425087ba76cfc426b6a3 (table test-table [id=172e93d1b0bf41659df0c5ae1a2bd256]) successfully deleted
I20260929 18:59:12.239004 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:12.238899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43705 (local address 127.0.0.1:51910)
0929 18:59:12.238938 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:12.238977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 18:59:12.239091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10803
W20260929 18:59:12.239108 10593 catalog_manager.cc:4584] TS 5625a6ee0a094b419245b9539a064263 (127.0.98.1:43705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc2f09a395eb425087ba76cfc426b6a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43705: connect: Connection refused (error 111)
I20260929 18:59:12.243942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10937
I20260929 18:59:12.248507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 10575
2026-09-29T18:59:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1857 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[11108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[11108](info): set up 2 sockets
Sep 29 18:59:12 dist-test-slave-gfsd krb5kdc[11108](info): commencing operation
krb5kdc: starting...
W20260929 18:59:13.292289 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 18:59:13 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:59:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 18:59:13.457831 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHoo0t8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/postgres -l logfile start
2026-09-29 18:59:15.944 UTC [11138] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:59:15.944 UTC [11138] LOG: listening on IPv4 address "127.0.98.20", port 41455
2026-09-29 18:59:15.949 UTC [11138] LOG: listening on Unix socket "/tmp/.s.PGSQL.41455"
2026-09-29 18:59:15.971 UTC [11141] LOG: database system was shut down at 2026-09-29 18:59:14 UTC
2026-09-29 18:59:15.975 UTC [11138] LOG: database system is ready to accept connections
I20260929 18:59:16.948330 392 mini_postgres.cc:96] Postgres bound to 41455
2026-09-29 18:59:16.954 UTC [11149] FATAL: database "slave" does not exist
127.0.98.20:41455 - accepting connections
I20260929 18:59:16.955171 392 mini_ranger.cc:161] Starting Ranger...
I20260929 18:59:16.969447 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:59:17.486163 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:59:17.486248 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin
2026-09-29 18:59:17,766 [I] DB FLAVOR :POSTGRES
2026-09-29 18:59:17,766 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:59:17,767 [I] Checking connection..
2026-09-29 18:59:17,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:59:18,009 [I] Checking connection passed.
2026-09-29 18:59:18,010 [I] --------- Verifying version history table ---------
2026-09-29 18:59:18,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:59:18,272 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:59:18,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:59:18,522 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:59:18,522 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:59:18,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:59:18.761 UTC [11213] WARNING: there is no transaction in progress
2026-09-29 18:59:18,768 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:59:18,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:59:19,029 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:59:19,029 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:59:19,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:59:19,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:59:19,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gfsd.c.gcp-upstream.internal','N') ;"
2026-09-29 18:59:19,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:59:20,001 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:59:20,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:59:20,244 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:59:20,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:59:20,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:59:20,728 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:59:20,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:59:21.840 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:21.864 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:21.892 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:21.913 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:21.938 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:21.977 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.046 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.054 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.070 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.080 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.091 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.111 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.122 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.128 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.135 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22.143 UTC [11321] WARNING: there is no transaction in progress
2026-09-29 18:59:22,914 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:59:22,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:59:23,145 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:59:23,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:59:23,415 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:59:23,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:59:23,641 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:59:23,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:59:23,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:59:24,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gfsd.c.gcp-upstream.internal';"
2026-09-29 18:59:24,382 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:59:24,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:59:24,610 [I] DB_PATCHES have already been applied
I20260929 18:59:24.616552 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHoo0t8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:59:24.616619 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 18:59:24.619658 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53285
Sep 29, 2026 6:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:59:25.793998 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.839s user 0.000s sys 0.009s
Sep 29, 2026 6:59:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:59:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 6:59:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:59:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:59:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:59:47.617223 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:59:47 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708387, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:59:47.706874 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:59:47.716161 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:59:48.602531 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:59:48.602680 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms
I20260929 18:59:48.602708 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41455
2026-09-29 18:59:48,779 [I] DB FLAVOR :POSTGRES
2026-09-29 18:59:48,779 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:59:48,779 [I] Checking connection
2026-09-29 18:59:48,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:59:49,047 [I] connection success
2026-09-29 18:59:49,047 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:59:49,047 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:59:49,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:59:49,276 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:59:49,276 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:59:49,276 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:59:49,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:59:49,552 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:59:50.019044 392 mini_ranger_kms.cc:310] Created kms service
I20260929 18:59:50.252338 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:59:50.295938 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:59:50 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708387, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:50.695434 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:59:50.695554 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:59:50.695578 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 18:59:50.699226 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11501
I20260929 18:59:50.699379 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48383
Sep 29, 2026 6:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:59:52.945979 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.239s user 0.012s sys 0.148s
I20260929 18:59:52.946130 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:59:52.946215 392 mini_ranger_kms.cc:401] 127.0.98.20:48383/kms/v1/keys
Sep 29 18:59:53 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708393, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:59:53 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708393, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:59:53 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708393, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:59:56 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:59:56.158396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33869
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48383/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:56.256093 11552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:56.256310 11552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:56.256335 11552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:56.257478 11552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:59:56.257525 11552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:56.257542 11552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:56.257556 11552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:59:56.257594 11552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:59:56.259167 11552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33869
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:56.259472 11552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:56.259722 11552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:59:56.265180 11552 server_base.cc:1034] running on GCE node
W20260929 18:59:56.265190 11558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:56.265192 11557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:56.265547 11560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:56.265825 11552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:56.266155 11552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:56.267361 11552 hybrid_clock.cc:648] HybridClock initialized: now 1790708396267333 us; error 41 us; skew 500 ppm
Sep 29 18:59:56 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:56.275516 11552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:59:56.276101 11552 webserver.cc:460] Webserver started at http://127.0.98.62:41295/ using document root <none> and password file <none>
I20260929 18:59:56.276367 11552 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:56.276428 11552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:56.276575 11552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:59:56 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:56.323201 11552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "bd9b91868d754035a0a892fe7c28b034"
format_stamp: "Formatted at 2026-09-29 18:59:56 on dist-test-slave-gfsd"
server_key: "e6bdca316970bb839f37bab4749d13ec"
server_key_iv: "8f0e45fa4a0ac9028270d77516336f86"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:56.323712 11552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd9b91868d754035a0a892fe7c28b034"
format_stamp: "Formatted at 2026-09-29 18:59:56 on dist-test-slave-gfsd"
server_key: "e6bdca316970bb839f37bab4749d13ec"
server_key_iv: "8f0e45fa4a0ac9028270d77516336f86"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:56.325276 11552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:59:56.336879 11567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:56.337087 11552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:59:56.337225 11552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "bd9b91868d754035a0a892fe7c28b034"
format_stamp: "Formatted at 2026-09-29 18:59:56 on dist-test-slave-gfsd"
server_key: "e6bdca316970bb839f37bab4749d13ec"
server_key_iv: "8f0e45fa4a0ac9028270d77516336f86"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:56.337327 11552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:56.355537 11552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:56.357597 11552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:56.357766 11552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:56.362975 11552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33869
I20260929 18:59:56.362999 11619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33869 every 8 connection(s)
I20260929 18:59:56.363435 11552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:59:56.364068 11620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:56.365187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 11552
I20260929 18:59:56.365288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 18:59:56.365497 392 external_mini_cluster.cc:1428] Setting key cc97e01b435a91a9b51d909e5eb739c6
I20260929 18:59:56.366685 11620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: Bootstrap starting.
I20260929 18:59:56.367447 11620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:56.367936 11620 log.cc:824] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:56.368731 11620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: No bootstrap required, opened a new log
I20260929 18:59:56.370961 11620 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } }
I20260929 18:59:56.371125 11620 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:56.371187 11620 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9b91868d754035a0a892fe7c28b034, State: Initialized, Role: FOLLOWER
I20260929 18:59:56.371353 11620 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } }
I20260929 18:59:56.371454 11620 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:59:56.371778 11620 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:59:56.371865 11620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:56.372383 11620 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } }
I20260929 18:59:56.372498 11620 leader_election.cc:302] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9b91868d754035a0a892fe7c28b034; no voters:
I20260929 18:59:56.372668 11620 leader_election.cc:288] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:59:56.372702 11625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:56.372996 11620 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:59:56.373000 11625 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [term 1 LEADER]: Becoming Leader. State: Replica: bd9b91868d754035a0a892fe7c28b034, State: Running, Role: LEADER
I20260929 18:59:56.373170 11625 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } }
Sep 29 18:59:56 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:56.373472 11625 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9b91868d754035a0a892fe7c28b034. Latest consensus state: current_term: 1 leader_uuid: "bd9b91868d754035a0a892fe7c28b034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } } }
I20260929 18:59:56.373544 11625 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:56.373813 11627 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9b91868d754035a0a892fe7c28b034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9b91868d754035a0a892fe7c28b034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33869 } } }
I20260929 18:59:56.373934 11627 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034 [sys.catalog]: This master's current role is: LEADER
I20260929 18:59:56.374095 11632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:59:56.374651 11632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:59:56.375412 11620 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:59:56.381228 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:56.366016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.98.62:33869)
0929 18:59:56.366191 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:56.366200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:56.366231 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:59:56.366778 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:56.366788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:56.366856 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:56.367017 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:56.367886 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:56.369331 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:56.370481 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:56.371114 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:56.371580 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:56.374163 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:56.374167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:56.374168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:56.374179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:56.377998 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:56.378522 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:56.378528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:56.378529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:56.378730 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:56.378990 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:56.378990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:56.378991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:56.379632 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:56.380330 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:56.380420 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:59:56.380895 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260929 18:59:56.386673 11632 catalog_manager.cc:1347] Generated new cluster ID: 6ad3abc0df52411aacb8a52491c09601
I20260929 18:59:56.386797 11632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:59:56.408816 11632 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:59:56.409401 11632 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:59:56.417860 11632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: Generated new TSK 0
I20260929 18:59:56.418123 11632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:59:57.186925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:57.289067 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:57.289306 11665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:57.289346 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:57.290490 11665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:57.290560 11665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:57.290602 11665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:59:57.292125 11665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:57.292387 11665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:57.292614 11665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:57.298131 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.298409 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.298431 11673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:57.299690 11665 server_base.cc:1034] running on GCE node
I20260929 18:59:57.299881 11665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:57.300141 11665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:57.301319 11665 hybrid_clock.cc:648] HybridClock initialized: now 1790708397301298 us; error 38 us; skew 500 ppm
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:57.304488 11665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:59:57.304999 11665 webserver.cc:460] Webserver started at http://127.0.98.1:36299/ using document root <none> and password file <none>
I20260929 18:59:57.305236 11665 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:57.305305 11665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:57.305435 11665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:57.319898 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "335aefa447bc418da1bddd9c36359e44"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "3e2bb38d319c05d14f5f20e26eee37d7"
server_key_iv: "36abd7f9d90679d7beaaac65010b30c3"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.320359 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "335aefa447bc418da1bddd9c36359e44"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "3e2bb38d319c05d14f5f20e26eee37d7"
server_key_iv: "36abd7f9d90679d7beaaac65010b30c3"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.325706 11665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260929 18:59:57.351449 11682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.351751 11665 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260929 18:59:57.351866 11665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "335aefa447bc418da1bddd9c36359e44"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "3e2bb38d319c05d14f5f20e26eee37d7"
server_key_iv: "36abd7f9d90679d7beaaac65010b30c3"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.351974 11665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:57.381699 11665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:57.392976 11665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:57.393203 11665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:57.393532 11665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:57.393980 11665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:57.394093 11665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.394191 11665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:57.394259 11665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.401309 11665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35325
I20260929 18:59:57.401443 11795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35325 every 8 connection(s)
I20260929 18:59:57.401885 11665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:59:57.403717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 11665
I20260929 18:59:57.403863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:59:57.404122 392 external_mini_cluster.cc:1428] Setting key 140199a71bb62ffb65750ac844c41dfd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:59:57.423501 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.402710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39961 (local address 127.0.98.62:33869)
0929 18:59:57.403799 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.403801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.403810 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:57.403833 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.403834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.403872 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.403952 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:57.404711 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.405556 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.407313 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.407575 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.407669 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.419662 (+ 11993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:57.419664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:57.419664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:57.419673 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:57.421662 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.422306 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.422307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.422307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.422368 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.422646 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.422646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.422647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.422781 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:57.422826 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.423274 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.423390 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":76,"threads_started":1}
I20260929 18:59:57.424264 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.402671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33869 (local address 127.0.98.1:39961)
0929 18:59:57.402863 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.402902 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.403542 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.404039 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.404044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.404303 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:57.404601 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.404607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.405667 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.405668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.407085 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.407088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.407686 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.407687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.407823 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.409607 (+ 1784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:57.409618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:57.419518 (+ 9900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:57.421817 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.421834 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.421840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.422167 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.422438 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.422439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.422439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.422576 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.422875 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:57.422881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:57.423068 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.423848 (+ 780us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.424020 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260929 18:59:57.424996 11796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33869
I20260929 18:59:57.425108 11796 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:57.425307 11796 heartbeater.cc:511] Master 127.0.98.62:33869 requested a full tablet report, sending...
I20260929 18:59:57.425886 11584 ts_manager.cc:194] Registered new tserver with Master: 335aefa447bc418da1bddd9c36359e44 (127.0.98.1:35325)
I20260929 18:59:57.426785 11584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:59:57.442663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:59:57.533457 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:57.533679 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:57.533717 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:57.534932 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:57.535000 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:57.535044 11804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:59:57.536571 11804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:57.536840 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:57.537084 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:57.542213 11812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.542197 11809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.542194 11810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:57.542737 11804 server_base.cc:1034] running on GCE node
I20260929 18:59:57.542954 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:57.543208 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:57.544360 11804 hybrid_clock.cc:648] HybridClock initialized: now 1790708397544352 us; error 51 us; skew 500 ppm
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:57.546840 11804 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:59:57.547358 11804 webserver.cc:460] Webserver started at http://127.0.98.2:40573/ using document root <none> and password file <none>
I20260929 18:59:57.547588 11804 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:57.547654 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:57.547804 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:57.564567 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e9b4a7b218f401d805a506517d88e4d"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "e765b21d5bb6806e46d317ea651ea98e"
server_key_iv: "93abe135c2bdfa28f687754d650a74aa"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.565112 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e9b4a7b218f401d805a506517d88e4d"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "e765b21d5bb6806e46d317ea651ea98e"
server_key_iv: "93abe135c2bdfa28f687754d650a74aa"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.567004 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:59:57.585697 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.585974 11804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:59:57.586068 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "1e9b4a7b218f401d805a506517d88e4d"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "e765b21d5bb6806e46d317ea651ea98e"
server_key_iv: "93abe135c2bdfa28f687754d650a74aa"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.586159 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:57.601786 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:57.604113 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:57.604276 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:57.604511 11804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:57.604872 11804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:57.604925 11804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.604991 11804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:57.605021 11804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.612516 11804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42699
I20260929 18:59:57.612882 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:59:57.613485 11933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42699 every 8 connection(s)
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:57.618945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 11804
I20260929 18:59:57.619050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:59:57.619271 392 external_mini_cluster.cc:1428] Setting key cd4f9837719caa446cf93dc04f3483a4
I20260929 18:59:57.622488 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.613885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33787 (local address 127.0.98.62:33869)
0929 18:59:57.613927 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.613928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.614023 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:59:57.614536 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.614537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.614550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.614607 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:57.615150 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.615729 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.616696 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.616918 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.616995 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.619097 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:57.619098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:57.619099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:57.619106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:57.620858 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.621356 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.621357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.621358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.621416 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.621714 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.621715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.621715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.621850 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:57.621896 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.622269 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.622375 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:59:57.623096 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.613622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33869 (local address 127.0.98.2:33787)
0929 18:59:57.613786 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.613834 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.614434 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.614680 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.614684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.614868 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:57.615054 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.615058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.615828 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.615829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.616604 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.616606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.616999 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.617000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.617125 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.617644 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:57.617653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:57.618979 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:57.620951 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.620957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.620959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.621257 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.621493 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.621494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.621494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.621632 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.621942 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:57.621947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:57.622119 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.622828 (+ 709us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.622946 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260929 18:59:57.623555 11934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33869
I20260929 18:59:57.623679 11934 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:57.623914 11934 heartbeater.cc:511] Master 127.0.98.62:33869 requested a full tablet report, sending...
I20260929 18:59:57.624282 11584 ts_manager.cc:194] Registered new tserver with Master: 1e9b4a7b218f401d805a506517d88e4d (127.0.98.2:42699)
I20260929 18:59:57.624776 11584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33787
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:59:57.646389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:59:57.762897 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:59:57.763127 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:59:57.763164 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:59:57.764386 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:59:57.764451 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:59:57.764496 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:59:57.765885 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:59:57.766146 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:59:57.766377 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:59:57.771762 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.771762 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:59:57.772053 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:59:57.772774 11939 server_base.cc:1034] running on GCE node
I20260929 18:59:57.773020 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:59:57.773211 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:59:57.774341 11939 hybrid_clock.cc:648] HybridClock initialized: now 1790708397774330 us; error 32 us; skew 500 ppm
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:59:57.777365 11939 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:59:57.778062 11939 webserver.cc:460] Webserver started at http://127.0.98.3:35683/ using document root <none> and password file <none>
I20260929 18:59:57.778338 11939 fs_manager.cc:359] Metadata directory not provided
I20260929 18:59:57.778436 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:59:57.778594 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:57.799708 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe0b175f3ef246a996d0d92cfc10ae92"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "1e401689d1d2c8b005092b648304f0c4"
server_key_iv: "4465bf5d6fddca3b2baefe3bb157013e"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.800297 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe0b175f3ef246a996d0d92cfc10ae92"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "1e401689d1d2c8b005092b648304f0c4"
server_key_iv: "4465bf5d6fddca3b2baefe3bb157013e"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.802222 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:59:57.823403 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.823709 11939 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260929 18:59:57.823863 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "fe0b175f3ef246a996d0d92cfc10ae92"
format_stamp: "Formatted at 2026-09-29 18:59:57 on dist-test-slave-gfsd"
server_key: "1e401689d1d2c8b005092b648304f0c4"
server_key_iv: "4465bf5d6fddca3b2baefe3bb157013e"
server_key_version: "kuduclusterkey@0"
I20260929 18:59:57.824030 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:59:57.848685 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:59:57.851810 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:59:57.852021 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:59:57.852304 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:59:57.852685 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:59:57.852789 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.852927 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:59:57.852988 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:59:57.861574 11939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42015
I20260929 18:59:57.862077 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:59:57.862766 12067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42015 every 8 connection(s)
I20260929 18:59:57.863138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 11939
I20260929 18:59:57.863250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:59:57.863479 392 external_mini_cluster.cc:1428] Setting key 346a3ca3fbf8e29a2f23014ea92edaee
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:59:57.876415 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.864398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42443 (local address 127.0.98.62:33869)
0929 18:59:57.864448 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.864451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.865067 (+ 616us) server_negotiation.cc:407] Connection header received
0929 18:59:57.865774 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.865775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.865786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.865815 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:57.866629 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.867449 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.868940 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.869258 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.869298 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.872404 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:57.872405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:57.872406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:57.872413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:57.874154 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.874770 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.874772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.874772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.874830 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.875622 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.875623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.875624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.875765 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:57.875814 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.876219 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.876315 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 18:59:57.877401 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.864128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33869 (local address 127.0.98.3:42443)
0929 18:59:57.864949 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.864994 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.865653 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.866009 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.866013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.866250 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:59:57.866520 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.866527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.867558 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.867559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.868824 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.868827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.869404 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.869405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.869540 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.870129 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:59:57.870146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:59:57.872297 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:59:57.874261 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.874267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.874270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.874645 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.874900 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:59:57.874900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:59:57.874901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:59:57.875537 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:59:57.875865 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:59:57.875871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:59:57.876051 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.876742 (+ 691us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.877146 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":77,"threads_started":1}
I20260929 18:59:57.878194 12068 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33869
I20260929 18:59:57.878300 12068 heartbeater.cc:464] Registering TS with master...
I20260929 18:59:57.878485 12068 heartbeater.cc:511] Master 127.0.98.62:33869 requested a full tablet report, sending...
I20260929 18:59:57.878885 11584 ts_manager.cc:194] Registered new tserver with Master: fe0b175f3ef246a996d0d92cfc10ae92 (127.0.98.3:42015)
I20260929 18:59:57.879417 11584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42443
I20260929 18:59:57.886631 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:59:57.896159 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.888299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.0.98.62:33869)
0929 18:59:57.888340 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.888342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.888481 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:59:57.888599 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.888600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.888610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.888647 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:59:57.889392 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.890136 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.891231 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.891553 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.891626 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.893354 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:59:57.893355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:59:57.893355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:59:57.893362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:59:57.894980 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.895388 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.895390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.895390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.895454 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:59:57.895635 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:59:57.895636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:59:57.895636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:59:57.895746 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:59:57.895848 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.895982 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.896050 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:59:57.897890 11584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:59:57.899859 11584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:59:57.912612 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.0.1:53402)
0929 18:59:57.905187 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.905245 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.905317 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.905627 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.905630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.905632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:57.905696 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.905698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.909861 (+ 4163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.909862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.910931 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.910933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.912264 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.912265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.912372 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.912388 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.912454 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.912511 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":87,"threads_started":1}
I20260929 18:59:57.912910 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.0.98.1:35325)
0929 18:59:57.904676 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.904682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.905404 (+ 722us) server_negotiation.cc:407] Connection header received
0929 18:59:57.905451 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.905453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.905471 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.905540 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:57.905781 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.909731 (+ 3950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.911038 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.912102 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.912163 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.912231 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.912787 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.912828 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260929 18:59:57.913250 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42699 (local address 127.0.0.1:40170)
0929 18:59:57.906988 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.906998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.907046 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.907352 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.907352 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.907354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:57.907395 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.907397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.908620 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.908621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.910888 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.910891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.912948 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.912949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.913090 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.913094 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.913140 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.913181 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260929 18:59:57.913558 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.0.98.2:42699)
0929 18:59:57.905123 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.905128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.907105 (+ 1977us) server_negotiation.cc:407] Connection header received
0929 18:59:57.907149 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.907151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.907167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.907251 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:57.907475 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.908520 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.911009 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.911758 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.911844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.912213 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.913432 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.913473 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":94,"threads_started":1}
I20260929 18:59:57.913806 11711 tablet_service.cc:1467] Processing CreateTablet for tablet ff4063039f114893a21e50b7b2dd2f39 (DEFAULT_TABLE table=test-table [id=7976a31abbaf46438423e4e258b8f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:57.914203 11711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4063039f114893a21e50b7b2dd2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:57.914160 11868 tablet_service.cc:1467] Processing CreateTablet for tablet ff4063039f114893a21e50b7b2dd2f39 (DEFAULT_TABLE table=test-table [id=7976a31abbaf46438423e4e258b8f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:57.914521 11868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4063039f114893a21e50b7b2dd2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:57.917877 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.0.1:41420)
0929 18:59:57.912270 (+ 7499us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.912280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.912328 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.912661 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.912661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.912663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:57.912870 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.912872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.914973 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.914974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.916414 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.916417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.917481 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.917482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.917547 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.917552 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.917605 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.917641 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7483,"thread_start_us":73,"threads_started":1}
I20260929 18:59:57.918154 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.904880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.0.98.3:42015)
0929 18:59:57.912450 (+ 7570us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.912455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.912464 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:57.912502 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.912503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.912518 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.912584 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:57.913821 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.914863 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.916534 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.917299 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.917375 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.917446 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.918035 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.918076 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7531,"thread_start_us":87,"threads_started":1}
I20260929 18:59:57.918349 12086 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: Bootstrap starting.
I20260929 18:59:57.918835 12002 tablet_service.cc:1467] Processing CreateTablet for tablet ff4063039f114893a21e50b7b2dd2f39 (DEFAULT_TABLE table=test-table [id=7976a31abbaf46438423e4e258b8f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:59:57.919135 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4063039f114893a21e50b7b2dd2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:59:57.919997 12086 tablet_bootstrap.cc:654] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:57.920287 12086 log.cc:824] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:57.920596 12087 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Bootstrap starting.
I20260929 18:59:57.921519 12087 tablet_bootstrap.cc:654] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:57.921772 12087 log.cc:824] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:57.922634 12090 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Bootstrap starting.
I20260929 18:59:57.923547 12090 tablet_bootstrap.cc:654] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Neither blocks nor log segments found. Creating new log.
I20260929 18:59:57.923887 12086 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: No bootstrap required, opened a new log
I20260929 18:59:57.923959 12086 ts_tablet_manager.cc:1397] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 18:59:57.923897 12090 log.cc:824] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Log is configured to *not* fsync() on all Append() calls
I20260929 18:59:57.924698 12090 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: No bootstrap required, opened a new log
I20260929 18:59:57.924777 12090 ts_tablet_manager.cc:1397] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:59:57.926642 12090 raft_consensus.cc:348] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.926666 12086 raft_consensus.cc:348] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.926815 12090 raft_consensus.cc:374] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:57.926834 12086 raft_consensus.cc:374] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:57.926853 12090 raft_consensus.cc:729] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b175f3ef246a996d0d92cfc10ae92, State: Initialized, Role: FOLLOWER
I20260929 18:59:57.926865 12086 raft_consensus.cc:729] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335aefa447bc418da1bddd9c36359e44, State: Initialized, Role: FOLLOWER
I20260929 18:59:57.926966 12090 consensus_queue.cc:260] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.926992 12086 consensus_queue.cc:260] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.927198 12090 ts_tablet_manager.cc:1428] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:59:57.927217 12086 ts_tablet_manager.cc:1428] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:59:57.927335 11796 heartbeater.cc:503] Master 127.0.98.62:33869 was elected leader, sending a full tablet report...
I20260929 18:59:57.927416 12068 heartbeater.cc:503] Master 127.0.98.62:33869 was elected leader, sending a full tablet report...
I20260929 18:59:57.928552 12087 tablet_bootstrap.cc:492] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: No bootstrap required, opened a new log
I20260929 18:59:57.928612 12087 ts_tablet_manager.cc:1397] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 18:59:57.930373 12087 raft_consensus.cc:348] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.930523 12087 raft_consensus.cc:374] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:59:57.930552 12087 raft_consensus.cc:729] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9b4a7b218f401d805a506517d88e4d, State: Initialized, Role: FOLLOWER
I20260929 18:59:57.930706 12087 consensus_queue.cc:260] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.931012 12087 ts_tablet_manager.cc:1428] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:59:57.931109 11934 heartbeater.cc:503] Master 127.0.98.62:33869 was elected leader, sending a full tablet report...
I20260929 18:59:57.943845 12092 raft_consensus.cc:482] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:59:57.943948 12092 raft_consensus.cc:504] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.944257 12092 leader_election.cc:288] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 335aefa447bc418da1bddd9c36359e44 (127.0.98.1:35325), 1e9b4a7b218f401d805a506517d88e4d (127.0.98.2:42699)
I20260929 18:59:57.948625 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.944482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.98.3:60549)
0929 18:59:57.944856 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.944868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.944916 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.945329 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.945330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.945332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:57.945401 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.945403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.946394 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.946395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.947602 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.947606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.948382 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.948383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.948458 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.948462 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.948513 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.948550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260929 18:59:57.949497 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.945046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60549 (local address 127.0.98.1:35325)
0929 18:59:57.945133 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.945135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.945144 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:57.945171 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.945172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.945183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.945238 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:57.945486 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.946299 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.947703 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.948265 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.948339 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.948349 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.949394 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.949432 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:59:57.949754 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4063039f114893a21e50b7b2dd2f39" candidate_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "335aefa447bc418da1bddd9c36359e44" is_pre_election: true
I20260929 18:59:57.949903 11738 raft_consensus.cc:2454] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0b175f3ef246a996d0d92cfc10ae92 in term 0.
I20260929 18:59:57.950204 11955 leader_election.cc:302] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335aefa447bc418da1bddd9c36359e44, fe0b175f3ef246a996d0d92cfc10ae92; no voters:
I20260929 18:59:57.950486 12092 raft_consensus.cc:2781] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:59:57.950543 12092 raft_consensus.cc:482] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:59:57.950567 12092 raft_consensus.cc:3037] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:57.950707 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.945092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48267 (local address 127.0.98.2:42699)
0929 18:59:57.945295 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.945297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.947478 (+ 2181us) server_negotiation.cc:407] Connection header received
0929 18:59:57.947570 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.947571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.947582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.947608 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:59:57.947831 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.948520 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.949848 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.950353 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.950394 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.950402 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.950613 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.950646 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260929 18:59:57.950899 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.944588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42699 (local address 127.0.98.3:48267)
0929 18:59:57.947407 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:57.947419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:59:57.947501 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:59:57.947667 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:59:57.947667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:59:57.947669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:59:57.947744 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.947745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.948675 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.948676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.949742 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:59:57.949744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.950480 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.950481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:59:57.950542 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.950545 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:59:57.950774 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 18:59:57.950820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2802,"thread_start_us":90,"threads_started":1}
I20260929 18:59:57.951207 11880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4063039f114893a21e50b7b2dd2f39" candidate_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9b4a7b218f401d805a506517d88e4d" is_pre_election: true
I20260929 18:59:57.951349 11880 raft_consensus.cc:2454] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0b175f3ef246a996d0d92cfc10ae92 in term 0.
I20260929 18:59:57.951534 12092 raft_consensus.cc:504] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.951654 12092 leader_election.cc:288] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [CANDIDATE]: Term 1 election: Requested vote from peers 335aefa447bc418da1bddd9c36359e44 (127.0.98.1:35325), 1e9b4a7b218f401d805a506517d88e4d (127.0.98.2:42699)
I20260929 18:59:57.951781 11880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4063039f114893a21e50b7b2dd2f39" candidate_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9b4a7b218f401d805a506517d88e4d"
I20260929 18:59:57.951854 11880 raft_consensus.cc:3037] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:57.951880 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4063039f114893a21e50b7b2dd2f39" candidate_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "335aefa447bc418da1bddd9c36359e44"
I20260929 18:59:57.951946 11738 raft_consensus.cc:3037] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:59:57.952605 11880 raft_consensus.cc:2454] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0b175f3ef246a996d0d92cfc10ae92 in term 1.
I20260929 18:59:57.952762 11957 leader_election.cc:302] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9b4a7b218f401d805a506517d88e4d, fe0b175f3ef246a996d0d92cfc10ae92; no voters:
I20260929 18:59:57.952843 11738 raft_consensus.cc:2454] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0b175f3ef246a996d0d92cfc10ae92 in term 1.
I20260929 18:59:57.953043 12092 raft_consensus.cc:2781] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:59:57.953176 12092 raft_consensus.cc:686] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 1 LEADER]: Becoming Leader. State: Replica: fe0b175f3ef246a996d0d92cfc10ae92, State: Running, Role: LEADER
I20260929 18:59:57.953248 12092 consensus_queue.cc:237] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } }
I20260929 18:59:57.954072 11583 catalog_manager.cc:5495] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe0b175f3ef246a996d0d92cfc10ae92 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe0b175f3ef246a996d0d92cfc10ae92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 } health_report { overall_health: UNKNOWN } } }
I20260929 18:59:57.969887 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.966470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.3:42015)
0929 18:59:57.966534 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:59:57.966536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:59:57.966545 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:59:57.966579 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:59:57.966580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:59:57.966601 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:59:57.966670 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:59:57.966859 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.967787 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.968951 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:59:57.969240 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:59:57.969313 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:59:57.969508 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:59:57.969634 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:59:57.969743 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:59:57.969765 (+ 22us) server_negotiation.cc:299] Negotiation successful
0929 18:59:57.969788 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:59:57.974757 11880 raft_consensus.cc:1261] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 1 FOLLOWER]: Refusing update from remote peer fe0b175f3ef246a996d0d92cfc10ae92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:59:57.975124 12092 consensus_queue.cc:1035] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9b4a7b218f401d805a506517d88e4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:59:57.977533 11738 raft_consensus.cc:1261] T ff4063039f114893a21e50b7b2dd2f39 P 335aefa447bc418da1bddd9c36359e44 [term 1 FOLLOWER]: Refusing update from remote peer fe0b175f3ef246a996d0d92cfc10ae92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:59:57.977808 12092 consensus_queue.cc:1035] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "335aefa447bc418da1bddd9c36359e44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 18:59:57 dist-test-slave-gfsd krb5kdc[11108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:59:57.980327 12100 mvcc.cc:205] Tried to move back new op lower bound from 7334741598101377024 to 7334741598017056768. Current Snapshot: MvccSnapshot[applied={T|T < 7334741598101377024}]
I20260929 18:59:57.980659 12099 mvcc.cc:205] Tried to move back new op lower bound from 7334741598101377024 to 7334741598017056768. Current Snapshot: MvccSnapshot[applied={T|T < 7334741598101377024 or (T in {7334741598101377024})}]
I20260929 18:59:57.982903 11584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:59:57.982987 11584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:59:57.983716 11584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9b91868d754035a0a892fe7c28b034: Sending DeleteTablet for 3 replicas of tablet ff4063039f114893a21e50b7b2dd2f39
I20260929 18:59:57.984126 11711 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4063039f114893a21e50b7b2dd2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53402
I20260929 18:59:57.984342 11868 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4063039f114893a21e50b7b2dd2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40170
I20260929 18:59:57.984683 12002 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4063039f114893a21e50b7b2dd2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41420
I20260929 18:59:57.985184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 11665
I20260929 18:59:57.986820 12109 tablet_replica.cc:330] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: stopping tablet replica
I20260929 18:59:57.986960 12109 raft_consensus.cc:2229] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:59:57.987131 12109 raft_consensus.cc:2258] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:59:57.987653 12109 ts_tablet_manager.cc:1905] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:59:57.987839 12108 tablet_replica.cc:330] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: stopping tablet replica
I20260929 18:59:57.987993 12108 raft_consensus.cc:2229] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:59:57.988157 12108 raft_consensus.cc:2258] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:59:57.989982 12109 ts_tablet_manager.cc:1918] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:59:57.990082 12109 log.cc:1196] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/ff4063039f114893a21e50b7b2dd2f39
I20260929 18:59:57.990401 12108 ts_tablet_manager.cc:1905] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:59:57.990418 12109 ts_tablet_manager.cc:1939] T ff4063039f114893a21e50b7b2dd2f39 P fe0b175f3ef246a996d0d92cfc10ae92: Deleting consensus metadata
I20260929 18:59:57.990790 11569 catalog_manager.cc:4841] TS fe0b175f3ef246a996d0d92cfc10ae92 (127.0.98.3:42015): tablet ff4063039f114893a21e50b7b2dd2f39 (table test-table [id=7976a31abbaf46438423e4e258b8f5b8]) successfully deleted
I20260929 18:59:57.992030 12108 ts_tablet_manager.cc:1918] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:59:57.992110 12108 log.cc:1196] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/ff4063039f114893a21e50b7b2dd2f39
I20260929 18:59:57.992650 12108 ts_tablet_manager.cc:1939] T ff4063039f114893a21e50b7b2dd2f39 P 1e9b4a7b218f401d805a506517d88e4d: Deleting consensus metadata
I20260929 18:59:57.993062 11570 catalog_manager.cc:4841] TS 1e9b4a7b218f401d805a506517d88e4d (127.0.98.2:42699): tablet ff4063039f114893a21e50b7b2dd2f39 (table test-table [id=7976a31abbaf46438423e4e258b8f5b8]) successfully deleted
W20260929 18:59:57.998107 11955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35325 (error 108)
W20260929 18:59:57.998549 11568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35325 (error 108)
I20260929 18:59:57.999478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 11804
I20260929 18:59:58.000542 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:58.000431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.98.3:53875)
0929 18:59:58.000477 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:58.000510 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:59:58.001272 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:59:57.999350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.0.1:53418)
0929 18:59:58.001186 (+ 1836us) negotiation.cc:107] Waiting for socket to connect
0929 18:59:58.001233 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1814}
W20260929 18:59:58.001444 11568 catalog_manager.cc:4584] TS 335aefa447bc418da1bddd9c36359e44 (127.0.98.1:35325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff4063039f114893a21e50b7b2dd2f39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35325: connect: Connection refused (error 111)
I20260929 18:59:58.009260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 11939
I20260929 18:59:58.015144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 11552
I20260929 18:59:58.025938 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:59:58.369787 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:59:58.369903 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:59:58.713873 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:59:58.714 UTC [11138] LOG: received smart shutdown request
2026-09-29 18:59:58.717 UTC [11138] LOG: background worker "logical replication launcher" (PID 11147) exited with exit code 1
2026-09-29 18:59:58.718 UTC [11142] LOG: shutting down
2026-09-29 18:59:58.880 UTC [11138] LOG: database system is shut down
2026-09-29T18:59:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46755 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:59:59 dist-test-slave-gfsd krb5kdc[12120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:59:59 dist-test-slave-gfsd krb5kdc[12120](info): set up 2 sockets
Sep 29 18:59:59 dist-test-slave-gfsd krb5kdc[12120](info): commencing operation
krb5kdc: starting...
W20260929 19:00:00.079370 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:00 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 19:00:00.245548 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHoo0t8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/postgres -l logfile start
2026-09-29 19:00:02.771 UTC [12150] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 19:00:02.771 UTC [12150] LOG: listening on IPv4 address "127.0.98.20", port 39387
2026-09-29 19:00:02.775 UTC [12150] LOG: listening on Unix socket "/tmp/.s.PGSQL.39387"
2026-09-29 19:00:02.802 UTC [12153] LOG: database system was shut down at 2026-09-29 19:00:01 UTC
2026-09-29 19:00:02.806 UTC [12150] LOG: database system is ready to accept connections
I20260929 19:00:03.759184 392 mini_postgres.cc:96] Postgres bound to 39387
2026-09-29 19:00:03.765 UTC [12161] FATAL: database "slave" does not exist
127.0.98.20:39387 - accepting connections
I20260929 19:00:03.765528 392 mini_ranger.cc:161] Starting Ranger...
I20260929 19:00:03.779811 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 19:00:04.284078 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 19:00:04.284171 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin
2026-09-29 19:00:04,549 [I] DB FLAVOR :POSTGRES
2026-09-29 19:00:04,550 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:00:04,550 [I] Checking connection..
2026-09-29 19:00:04,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 19:00:04,810 [I] Checking connection passed.
2026-09-29 19:00:04,810 [I] --------- Verifying version history table ---------
2026-09-29 19:00:04,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:00:05,074 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:00:05,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:00:05,326 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:00:05,326 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 19:00:05,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 19:00:05.577 UTC [12225] WARNING: there is no transaction in progress
2026-09-29 19:00:05,583 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 19:00:05,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:00:05,823 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 19:00:05,823 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:00:05,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 19:00:06,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 19:00:06,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gfsd.c.gcp-upstream.internal','N') ;"
2026-09-29 19:00:06,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:00:06,824 [I] Table x_portal_user does not exist in database ranger
2026-09-29 19:00:06,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:00:07,075 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 19:00:07,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:00:07,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:00:07,579 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 19:00:07,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 19:00:08.550 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.567 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.583 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.598 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.616 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.634 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.688 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.695 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.709 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.718 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.728 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.741 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.751 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.757 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.764 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:08.771 UTC [12333] WARNING: there is no transaction in progress
2026-09-29 19:00:09,412 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 19:00:09,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 19:00:09,641 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 19:00:09,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:00:09,866 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 19:00:09,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:00:10,116 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 19:00:10,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:00:10,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:00:10,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gfsd.c.gcp-upstream.internal';"
2026-09-29 19:00:10,886 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 19:00:10,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 19:00:11,144 [I] DB_PATCHES have already been applied
I20260929 19:00:11.149776 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHoo0t8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 19:00:11.149847 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 19:00:11.152818 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51823
Sep 29, 2026 7:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 7:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 7:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 7:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 7:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 19:00:12.262399 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.497s user 0.003s sys 0.006s
Sep 29, 2026 7:00:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 7:00:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 7:00:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 7:00:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 7:00:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 19:00:33.360896 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 19:00:33 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 19:00:33.448297 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 19:00:33.457577 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 19:00:34.389598 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 19:00:34.389694 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms
I20260929 19:00:34.389719 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39387
2026-09-29 19:00:34,558 [I] DB FLAVOR :POSTGRES
2026-09-29 19:00:34,558 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:00:34,558 [I] Checking connection
2026-09-29 19:00:34,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 19:00:34,797 [I] connection success
2026-09-29 19:00:34,798 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 19:00:34,798 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 19:00:34,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 19:00:35,043 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 19:00:35,043 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:00:35,043 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 19:00:35,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 19:00:35,303 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 19:00:35.754158 392 mini_ranger_kms.cc:310] Created kms service
I20260929 19:00:35.936403 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 19:00:35.978933 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 19:00:35 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:36.383107 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 19:00:36.383391 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHoo0t8/build/release/bin/postgresql.jar:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHoo0t8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 19:00:36.383425 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 19:00:36.386643 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12513
I20260929 19:00:36.386715 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55665
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 7:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 19:00:38 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:38.679737 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.231s user 0.004s sys 0.148s
I20260929 19:00:38.679855 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 19:00:38.679912 392 mini_ranger_kms.cc:401] 127.0.98.20:55665/kms/v1/keys
Sep 29 19:00:39 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:00:39 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708438, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:00:41 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:41.738444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35785
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:41.824311 12564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:41.824508 12564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:41.824530 12564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:41.825691 12564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:41.825732 12564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:41.825747 12564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:41.825759 12564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:41.825771 12564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:41.827499 12564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55665/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35785
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:41.827736 12564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:41.827975 12564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:41.834092 12570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:41.834170 12564 server_base.cc:1034] running on GCE node
W20260929 19:00:41.834126 12569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:41.834303 12572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:41.834640 12564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:41.834928 12564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:41.836089 12564 hybrid_clock.cc:648] HybridClock initialized: now 1790708441836068 us; error 39 us; skew 500 ppm
Sep 29 19:00:41 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:41.842837 12564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:41.843390 12564 webserver.cc:460] Webserver started at http://127.0.98.62:36445/ using document root <none> and password file <none>
I20260929 19:00:41.843641 12564 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:41.843707 12564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:41.843843 12564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:00:41 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:41.904294 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "82ff0a43b783498595cdfacf7872706b"
format_stamp: "Formatted at 2026-09-29 19:00:41 on dist-test-slave-gfsd"
server_key: "7d19896c5a254be8c1e7b18946ecc856"
server_key_iv: "baf39b6c0757e9798bd71559f32efebd"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:41.904736 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "82ff0a43b783498595cdfacf7872706b"
format_stamp: "Formatted at 2026-09-29 19:00:41 on dist-test-slave-gfsd"
server_key: "7d19896c5a254be8c1e7b18946ecc856"
server_key_iv: "baf39b6c0757e9798bd71559f32efebd"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:41.906584 12564 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 19:00:41.921705 12579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:41.921998 12564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:41.922142 12564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "82ff0a43b783498595cdfacf7872706b"
format_stamp: "Formatted at 2026-09-29 19:00:41 on dist-test-slave-gfsd"
server_key: "7d19896c5a254be8c1e7b18946ecc856"
server_key_iv: "baf39b6c0757e9798bd71559f32efebd"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:41.922255 12564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:41.937455 12564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:41.939893 12564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:41.940073 12564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:41.945433 12564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35785
I20260929 19:00:41.945504 12631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35785 every 8 connection(s)
I20260929 19:00:41.945866 12564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:41.946136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 12564
I20260929 19:00:41.946233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:41.946446 392 external_mini_cluster.cc:1428] Setting key 5733a346700f61c2ebcd9ba36cc6e27c
I20260929 19:00:41.946650 12632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:41.950294 12632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Bootstrap starting.
I20260929 19:00:41.950975 12632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:41.951297 12632 log.cc:824] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:41.952080 12632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: No bootstrap required, opened a new log
Sep 29 19:00:41 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:41.955259 12632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:41.955462 12632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:41.955524 12632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ff0a43b783498595cdfacf7872706b, State: Initialized, Role: FOLLOWER
I20260929 19:00:41.955662 12632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:41.955734 12632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:41.955760 12632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:41.955844 12632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:41.956496 12632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:41.956631 12632 leader_election.cc:302] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82ff0a43b783498595cdfacf7872706b; no voters:
I20260929 19:00:41.956822 12632 leader_election.cc:288] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:41.956875 12637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:41.956908 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:41.946967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.98.62:35785)
0929 19:00:41.947209 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:41.947220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:41.947260 (+ 40us) server_negotiation.cc:407] Connection header received
0929 19:00:41.947740 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:41.947747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:41.947794 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:41.947946 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:41.948559 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:41.949454 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:41.950246 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:41.950727 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:41.950853 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:41.953645 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:41.953649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:41.953650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:41.953662 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:41.955272 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:41.955686 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:41.955689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:41.955689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:41.955832 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:41.956048 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:41.956049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:41.956049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:41.956294 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:41.956465 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:41.956532 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:00:41.956646 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":151,"threads_started":1}
I20260929 19:00:41.957098 12637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 LEADER]: Becoming Leader. State: Replica: 82ff0a43b783498595cdfacf7872706b, State: Running, Role: LEADER
I20260929 19:00:41.957115 12632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:41.957950 12637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:41.958647 12639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82ff0a43b783498595cdfacf7872706b. Latest consensus state: current_term: 1 leader_uuid: "82ff0a43b783498595cdfacf7872706b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260929 19:00:41.958748 12639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:41.958806 12632 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:00:41.958954 12638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82ff0a43b783498595cdfacf7872706b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260929 19:00:41.959048 12638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:41.959396 12651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:41.959860 12651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:41.965196 12651 catalog_manager.cc:1347] Generated new cluster ID: 64810fcca8ef4e71beb639534413c6d1
I20260929 19:00:41.965329 12651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:00:41.973903 12651 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:41.974659 12651 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:41.993374 12651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Generated new TSK 0
I20260929 19:00:41.993731 12651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:42.735335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:42.821564 12677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:42.821789 12677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:42.821827 12677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:42.823062 12677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:42.823139 12677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:42.823184 12677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:42.824779 12677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:42.825016 12677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:42.825248 12677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:42.830998 12683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:42.831107 12677 server_base.cc:1034] running on GCE node
W20260929 19:00:42.831216 12685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:42.831336 12682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:42.831544 12677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:42.831796 12677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:42.832955 12677 hybrid_clock.cc:648] HybridClock initialized: now 1790708442832942 us; error 32 us; skew 500 ppm
Sep 29 19:00:42 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:42.835680 12677 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:42.836131 12677 webserver.cc:460] Webserver started at http://127.0.98.1:42153/ using document root <none> and password file <none>
I20260929 19:00:42.836304 12677 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:42.836337 12677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:42.836422 12677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:00:42 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:42.850867 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc0c9ea314994493ac53ded39c633d52"
format_stamp: "Formatted at 2026-09-29 19:00:42 on dist-test-slave-gfsd"
server_key: "425290a273e1f3e5295f6cce824fa5bd"
server_key_iv: "ff622a606fe86cc3512a05ca2cb87d23"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:42.851410 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc0c9ea314994493ac53ded39c633d52"
format_stamp: "Formatted at 2026-09-29 19:00:42 on dist-test-slave-gfsd"
server_key: "425290a273e1f3e5295f6cce824fa5bd"
server_key_iv: "ff622a606fe86cc3512a05ca2cb87d23"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:42.853348 12677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:42.867638 12692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:42.867866 12677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:42.867960 12677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "cc0c9ea314994493ac53ded39c633d52"
format_stamp: "Formatted at 2026-09-29 19:00:42 on dist-test-slave-gfsd"
server_key: "425290a273e1f3e5295f6cce824fa5bd"
server_key_iv: "ff622a606fe86cc3512a05ca2cb87d23"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:42.868063 12677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:42.903487 12677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:42.906018 12677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:42.906190 12677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:42.906443 12677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:42.906876 12677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:42.906932 12677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:42.906965 12677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:42.906984 12677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:42.914469 12677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41341
I20260929 19:00:42.914981 12677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:42.915539 12807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41341 every 8 connection(s)
I20260929 19:00:42.921514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 12677
I20260929 19:00:42.921653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:42.921857 392 external_mini_cluster.cc:1428] Setting key 6878ba8859cbd9cf037546e4a8658f97
Sep 29 19:00:42 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:42.930115 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:42.915898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.1:56497)
0929 19:00:42.916143 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:42.916184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:42.916837 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:42.919953 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:42.919957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:42.920174 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:42.920372 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:42.920377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:42.921404 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:42.921405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:42.922849 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:42.922852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:42.923531 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:42.923533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:42.923685 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:42.924339 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:42.924350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:42.926121 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:42.928377 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:42.928384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:42.928386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:42.928710 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:42.928940 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:42.928941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:42.928941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:42.929080 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:42.929383 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:42.929389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:42.929569 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 19:00:42.929699 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 19:00:42.929871 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260929 19:00:42.930691 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:42.915911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56497 (local address 127.0.98.62:35785)
0929 19:00:42.919714 (+ 3803us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:42.919716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:42.919727 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:42.919756 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:42.919756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:42.919770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:42.919853 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:42.920480 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:42.921285 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:42.922936 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:42.923390 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:42.923454 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:42.926221 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:42.926223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:42.926223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:42.926231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:42.928275 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:42.928800 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:42.928801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:42.928801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:42.928868 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:42.929154 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:42.929154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:42.929155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:42.929293 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:42.929346 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:42.930488 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0929 19:00:42.930580 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3765,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:00:42.931528 12808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:42.931650 12808 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:42.931843 12808 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:42.932401 12596 ts_manager.cc:194] Registered new tserver with Master: cc0c9ea314994493ac53ded39c633d52 (127.0.98.1:41341)
I20260929 19:00:42.933303 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:42.952172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:43.057129 12814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:43.057358 12814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:43.057446 12814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:43.058835 12814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:43.058895 12814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:43.058940 12814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:43.060560 12814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:43.060827 12814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:43.061048 12814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:43.066238 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.066362 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.066246 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.067356 12814 server_base.cc:1034] running on GCE node
I20260929 19:00:43.067572 12814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:43.067860 12814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:43.069024 12814 hybrid_clock.cc:648] HybridClock initialized: now 1790708443069004 us; error 37 us; skew 500 ppm
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.071902 12814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:43.072387 12814 webserver.cc:460] Webserver started at http://127.0.98.2:34051/ using document root <none> and password file <none>
I20260929 19:00:43.072667 12814 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:43.072746 12814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:43.072870 12814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:43.084201 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9e9af98e0d34bfc89fb6da183e0e322"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "f70f9e950bc0d7df3364e5569675c75b"
server_key_iv: "7a059dc77dbc16897f1758b529dd084f"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.084720 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9e9af98e0d34bfc89fb6da183e0e322"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "f70f9e950bc0d7df3364e5569675c75b"
server_key_iv: "7a059dc77dbc16897f1758b529dd084f"
server_key_version: "kuduclusterkey@0"
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.087713 12814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 19:00:43.103901 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.104192 12814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 19:00:43.104328 12814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "b9e9af98e0d34bfc89fb6da183e0e322"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "f70f9e950bc0d7df3364e5569675c75b"
server_key_iv: "7a059dc77dbc16897f1758b529dd084f"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.104453 12814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:43.153939 12814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:43.156402 12814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:43.156569 12814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:43.156840 12814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:43.157260 12814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:43.157353 12814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.157400 12814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:43.157419 12814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.164847 12814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46577
I20260929 19:00:43.164920 12944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46577 every 8 connection(s)
I20260929 19:00:43.165293 12814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:43.168587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 12814
I20260929 19:00:43.168720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:43.168912 392 external_mini_cluster.cc:1428] Setting key dd25b4bf21eafdf5194ecf7cbc5fed71
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:43.176453 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.166060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54371 (local address 127.0.98.62:35785)
0929 19:00:43.166107 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:43.166109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:43.166324 (+ 215us) server_negotiation.cc:407] Connection header received
0929 19:00:43.166977 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:43.166978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:43.166987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:43.167032 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:43.167819 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.168327 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.169800 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.170097 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.170178 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.172842 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:43.172844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:43.172844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:43.172851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:43.174710 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.175213 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.175214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.175215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.175311 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.175617 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.175618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.175618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.175769 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:43.175833 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:43.176250 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 19:00:43.176346 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:00:43.177201 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.166021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.2:54371)
0929 19:00:43.166190 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:43.166233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:43.166852 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:43.167106 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:43.167111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:43.167417 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:43.167705 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.167712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.168433 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.168434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.169655 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.169658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.170227 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.170228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.170380 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.170998 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:43.171009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:43.172745 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:43.174808 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.174816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.174819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.175115 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.175391 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.175391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.175392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.175534 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.175886 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:43.175892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:43.176074 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 19:00:43.176785 (+ 711us) client_negotiation.cc:240] Negotiation successful
0929 19:00:43.176957 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 19:00:43.179612 12945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:43.179725 12945 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:43.179934 12945 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:43.180299 12596 ts_manager.cc:194] Registered new tserver with Master: b9e9af98e0d34bfc89fb6da183e0e322 (127.0.98.2:46577)
I20260929 19:00:43.180819 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:43.198923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:43.320184 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:43.320474 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:43.320516 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:43.322280 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:43.322342 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:43.322384 12951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:43.324553 12951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:43.324813 12951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:43.325114 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:43.331748 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.332006 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.332268 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.333500 12951 server_base.cc:1034] running on GCE node
I20260929 19:00:43.333798 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:43.334060 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:43.335222 12951 hybrid_clock.cc:648] HybridClock initialized: now 1790708443335189 us; error 48 us; skew 500 ppm
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.338518 12951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:43.339109 12951 webserver.cc:460] Webserver started at http://127.0.98.3:42033/ using document root <none> and password file <none>
I20260929 19:00:43.339484 12951 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:43.339560 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:43.339747 12951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:43.351765 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "88d460adc30d484680db88fe312afb7c"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "636eb59298cdc6e427271d7980dc58ec"
server_key_iv: "54b7b71122be596296b39d9711075708"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.352291 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88d460adc30d484680db88fe312afb7c"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "636eb59298cdc6e427271d7980dc58ec"
server_key_iv: "54b7b71122be596296b39d9711075708"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.353941 12951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:43.367432 12966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.367657 12951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:43.367743 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "88d460adc30d484680db88fe312afb7c"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "636eb59298cdc6e427271d7980dc58ec"
server_key_iv: "54b7b71122be596296b39d9711075708"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.367828 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:43.382831 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:43.385142 12951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:43.385317 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:43.385560 12951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:43.385948 12951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:43.385982 12951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.386006 12951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:43.386053 12951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.392450 12951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40331
I20260929 19:00:43.392627 13079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40331 every 8 connection(s)
I20260929 19:00:43.392871 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:43.397225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 12951
I20260929 19:00:43.397346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:43.397526 392 external_mini_cluster.cc:1428] Setting key 49449fb8b2e7ecce0d0d3753aaf672c6
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:43.404737 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.394072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39175 (local address 127.0.98.62:35785)
0929 19:00:43.394125 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:43.394128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:43.394379 (+ 251us) server_negotiation.cc:407] Connection header received
0929 19:00:43.394860 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:43.394861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:43.394875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:43.394949 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:43.395742 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.396458 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.398531 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.398797 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.398835 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.401378 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:43.401380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:43.401380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:43.401387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:43.403101 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.403646 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.403647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.403648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.403706 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.403986 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.403987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.403988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.404131 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:43.404184 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:43.404560 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 19:00:43.404643 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:00:43.405479 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.394033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.3:39175)
0929 19:00:43.394241 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:43.394281 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:43.394758 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:43.395049 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:43.395053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:43.395357 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:43.395630 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.395636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.396571 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.396572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.398430 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.398432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.398925 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.398926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.399056 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.399773 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:43.399783 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:43.401273 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:43.403221 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.403228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.403230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.403538 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.403780 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.403781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.403781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.403910 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.404218 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:43.404224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:43.404395 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 19:00:43.405077 (+ 682us) client_negotiation.cc:240] Negotiation successful
0929 19:00:43.405250 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260929 19:00:43.405938 13080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:43.406066 13080 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:43.406271 13080 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:43.406682 12596 ts_manager.cc:194] Registered new tserver with Master: 88d460adc30d484680db88fe312afb7c (127.0.98.3:40331)
I20260929 19:00:43.407152 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39175
I20260929 19:00:43.411139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:43.412004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 12677
I20260929 19:00:43.419559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 12814
I20260929 19:00:43.425221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 12951
I20260929 19:00:43.431262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 12564
I20260929 19:00:43.435873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--webserver_interface=127.0.98.62
--webserver_port=36445
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35785
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:43.517503 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:43.517709 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:43.517730 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:43.518769 13083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:43.518801 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:43.518814 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:43.518827 13083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:43.518838 13083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:43.520424 13083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35785
--ranger_config_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:43.520601 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:43.520785 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:43.525755 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.525751 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.525779 13083 server_base.cc:1034] running on GCE node
W20260929 19:00:43.525751 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.526191 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:43.526494 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:43.527659 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790708443527653 us; error 35 us; skew 500 ppm
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.530121 13083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:43.530627 13083 webserver.cc:460] Webserver started at http://127.0.98.62:36445/ using document root <none> and password file <none>
I20260929 19:00:43.530838 13083 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:43.530911 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:43.572851 13083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:43.573742 13099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.573904 13083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:43.574003 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "82ff0a43b783498595cdfacf7872706b"
format_stamp: "Formatted at 2026-09-29 19:00:41 on dist-test-slave-gfsd"
server_key: "7d19896c5a254be8c1e7b18946ecc856"
server_key_iv: "baf39b6c0757e9798bd71559f32efebd"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.574368 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:43.592844 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:43.594925 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:43.595044 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:43.599424 13083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35785
I20260929 19:00:43.599467 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35785 every 8 connection(s)
I20260929 19:00:43.599836 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:43.601004 13152 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:00:43.601382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13083
I20260929 19:00:43.601667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41341
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35785
--webserver_port=42153
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:43.601858 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Bootstrap starting.
I20260929 19:00:43.609683 13152 log.cc:824] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:43.611752 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:00:43.612165 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Bootstrap complete.
I20260929 19:00:43.614642 13152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:43.614912 13152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ff0a43b783498595cdfacf7872706b, State: Initialized, Role: FOLLOWER
I20260929 19:00:43.615103 13152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:43.615185 13152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:43.615238 13152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:43.615312 13152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:00:43.616319 13152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:43.616437 13152 leader_election.cc:302] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82ff0a43b783498595cdfacf7872706b; no voters:
I20260929 19:00:43.616638 13152 leader_election.cc:288] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:00:43.616680 13156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:00:43.616786 13156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [term 2 LEADER]: Becoming Leader. State: Replica: 82ff0a43b783498595cdfacf7872706b, State: Running, Role: LEADER
I20260929 19:00:43.616914 13152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:43.616986 13156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260929 19:00:43.617244 13156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82ff0a43b783498595cdfacf7872706b. Latest consensus state: current_term: 2 leader_uuid: "82ff0a43b783498595cdfacf7872706b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260929 19:00:43.617482 13156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:43.617403 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82ff0a43b783498595cdfacf7872706b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ff0a43b783498595cdfacf7872706b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260929 19:00:43.617861 13165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:43.617887 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:43.618392 13165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:43.618510 13165 catalog_manager.cc:1259] Loaded cluster ID: 64810fcca8ef4e71beb639534413c6d1
I20260929 19:00:43.618587 13165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:00:43.618808 13152 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:00:43.621872 13165 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:43.622514 13165 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Loaded TSK: 0
I20260929 19:00:43.622812 13165 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:00:43.714607 13154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:43.714895 13154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:43.714931 13154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:43.716740 13154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:43.716809 13154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:43.716849 13154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:43.718884 13154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42153
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:43.719164 13154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:43.719494 13154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:43.725996 13190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.726342 13192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.727039 13189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.727504 13154 server_base.cc:1034] running on GCE node
I20260929 19:00:43.727749 13154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:43.728029 13154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:43.729235 13154 hybrid_clock.cc:648] HybridClock initialized: now 1790708443729200 us; error 41 us; skew 500 ppm
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.732579 13154 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:43.733170 13154 webserver.cc:460] Webserver started at http://127.0.98.1:42153/ using document root <none> and password file <none>
I20260929 19:00:43.733470 13154 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:43.733592 13154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:43.756872 13154 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:43.759713 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.759914 13154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 19:00:43.759989 13154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "cc0c9ea314994493ac53ded39c633d52"
format_stamp: "Formatted at 2026-09-29 19:00:42 on dist-test-slave-gfsd"
server_key: "425290a273e1f3e5295f6cce824fa5bd"
server_key_iv: "ff622a606fe86cc3512a05ca2cb87d23"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.760416 13154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:43.770784 13154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:43.772715 13154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:43.772874 13154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:43.773168 13154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:43.773532 13154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:43.773583 13154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.773615 13154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:43.773644 13154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.780969 13154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41341
I20260929 19:00:43.781073 13313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41341 every 8 connection(s)
I20260929 19:00:43.781440 13154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:43.787364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13154
I20260929 19:00:43.787715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46577
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35785
--webserver_port=34051
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:43.794121 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.782281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47455 (local address 127.0.98.62:35785)
0929 19:00:43.782476 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:43.782485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:43.782720 (+ 235us) server_negotiation.cc:407] Connection header received
0929 19:00:43.783372 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:43.783378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:43.783429 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:43.783567 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:43.784137 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.785021 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.786045 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.786474 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.786608 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.790470 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:43.790475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:43.790475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:43.790487 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:43.792201 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.792716 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.792720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.792720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.792826 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:43.793094 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:43.793095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:43.793095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:43.793386 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:43.793480 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:43.793791 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 19:00:43.793902 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260929 19:00:43.794133 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.782224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.1:47455)
0929 19:00:43.782561 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:43.782599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:43.783113 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:43.783612 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:43.783615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:43.783818 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:43.783998 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.784002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.785150 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.785152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.785939 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.785941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:43.786600 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:43.786602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:43.786748 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:43.789012 (+ 2264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:43.789021 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:43.790353 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:43.792307 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.792313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.792315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.792568 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.792898 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:43.792899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:43.792900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:43.793011 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:43.793458 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:43.793464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:43.793627 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:00:43.793738 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 19:00:43.793902 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":91,"threads_started":1}
I20260929 19:00:43.795928 13314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:43.796028 13314 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:43.796258 13314 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:43.797128 13314 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:00:43.797211 13314 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:43.797276 13314 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:43.797587 13314 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:43.797670 13314 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:43.797969 13314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:00:43.907409 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:43.907672 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:43.907711 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:43.909415 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:43.909477 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:43.909519 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:43.911702 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34051
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:43.911942 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:43.912276 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:43.918694 13324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.918694 13326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:43.919417 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:43.920544 13318 server_base.cc:1034] running on GCE node
I20260929 19:00:43.920742 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:43.920993 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:43.922206 13318 hybrid_clock.cc:648] HybridClock initialized: now 1790708443922188 us; error 40 us; skew 500 ppm
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:43.925412 13318 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:43.925978 13318 webserver.cc:460] Webserver started at http://127.0.98.2:34051/ using document root <none> and password file <none>
I20260929 19:00:43.926359 13318 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:43.926455 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:00:43 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:43.948204 13318 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260929 19:00:43.952075 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.952273 13318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 19:00:43.952365 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "b9e9af98e0d34bfc89fb6da183e0e322"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "f70f9e950bc0d7df3364e5569675c75b"
server_key_iv: "7a059dc77dbc16897f1758b529dd084f"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:43.952857 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:43.984525 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:43.986732 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:43.986896 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:43.987227 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:43.987630 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:43.987703 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.987746 13318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:43.987773 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:43.993577 13318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46577
I20260929 19:00:43.993639 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46577 every 8 connection(s)
I20260929 19:00:43.994114 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:43.994285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13318
I20260929 19:00:43.994623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40331
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35785
--webserver_port=42033
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:00:44 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:44.008721 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.996083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32941 (local address 127.0.98.62:35785)
0929 19:00:43.996142 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:43.996144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:43.996291 (+ 147us) server_negotiation.cc:407] Connection header received
0929 19:00:43.996990 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:43.996992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:43.997005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:43.997038 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:43.999398 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.000174 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.001764 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.002064 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.002106 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:44.004986 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:44.004988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:44.004989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:44.004997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:44.007046 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.007613 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.007614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.007614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.007676 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.007980 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.007980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.007981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.008120 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:44.008182 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:44.008540 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 19:00:44.008620 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:00:44.008882 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:43.995769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.2:32941)
0929 19:00:43.996150 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:43.996196 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:43.996857 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:43.997218 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:43.997223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:43.997453 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:43.997744 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:43.997751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.000312 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.000313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:44.001658 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:44.001661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.002207 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.002208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:44.002356 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:44.002981 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:44.002991 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:44.004862 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:44.007143 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:44.007149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:44.007152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:44.007523 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:44.007749 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:44.007750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:44.007751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:44.007896 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:44.008193 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:44.008200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:44.008385 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 19:00:44.008496 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 19:00:44.008650 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":98,"threads_started":1}
I20260929 19:00:44.009585 13448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:44.009692 13448 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.009886 13448 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:44.010509 13448 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:00:44.010587 13448 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.010654 13448 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:44.010982 13448 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.011054 13448 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:44.011375 13448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:00:44.097744 13450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:44.097962 13450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:44.098001 13450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:44.099157 13450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:44.099226 13450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:44.099272 13450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:44.100893 13450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42033
--tserver_master_addrs=127.0.98.62:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:44.101146 13450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:44.101397 13450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:44.106921 13456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:44.107314 13457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:44.107757 13459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:44.108466 13450 server_base.cc:1034] running on GCE node
I20260929 19:00:44.108726 13450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:44.109004 13450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:44.110162 13450 hybrid_clock.cc:648] HybridClock initialized: now 1790708444110133 us; error 41 us; skew 500 ppm
Sep 29 19:00:44 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:44.112864 13450 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:44.113335 13450 webserver.cc:460] Webserver started at http://127.0.98.3:42033/ using document root <none> and password file <none>
I20260929 19:00:44.113580 13450 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:44.113670 13450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:00:44 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:44.145399 13450 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.001s sys 0.000s
I20260929 19:00:44.146384 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:44.146546 13450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:44.146629 13450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "88d460adc30d484680db88fe312afb7c"
format_stamp: "Formatted at 2026-09-29 19:00:43 on dist-test-slave-gfsd"
server_key: "636eb59298cdc6e427271d7980dc58ec"
server_key_iv: "54b7b71122be596296b39d9711075708"
server_key_version: "kuduclusterkey@0"
I20260929 19:00:44.147111 13450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:44.169751 13450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:44.172226 13450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:44.172384 13450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:44.172636 13450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:44.173020 13450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:44.173066 13450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:44.173095 13450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:44.173125 13450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:44.180227 13450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40331
I20260929 19:00:44.180356 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40331 every 8 connection(s)
I20260929 19:00:44.180714 13450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:44.183142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13450
Sep 29 19:00:44 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:44.191710 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:44.181706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50209 (local address 127.0.98.62:35785)
0929 19:00:44.181752 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:44.181754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:44.181765 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:44.182427 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:44.182428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:44.182441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:44.182492 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:44.183055 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.183783 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.185223 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.185418 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.185494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:44.187979 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:44.187981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:44.187982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:44.187989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:44.189712 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.190161 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.190163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.190163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.190225 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.190542 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.190544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.190544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.190682 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:44.190761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:44.191488 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 19:00:44.191615 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:00:44.191833 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:44.181474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35785 (local address 127.0.98.3:50209)
0929 19:00:44.181621 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:44.181672 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:44.182303 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:44.182561 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:44.182565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:44.182772 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:44.182958 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:44.182962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.183884 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.183885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:44.185118 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:44.185121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.185574 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.185575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:44.185727 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:44.186301 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:44.186311 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:44.187906 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:44.189819 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:44.189824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:44.189826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:44.190044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:44.190306 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:44.190307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:44.190307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:44.190461 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:44.190762 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:44.190769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:44.190953 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 19:00:44.191067 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 19:00:44.191566 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260929 19:00:44.192268 13582 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35785
I20260929 19:00:44.192386 13582 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.192602 13582 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:44.193331 13582 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:00:44.193401 13582 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.193486 13582 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:44.193864 13582 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.193948 13582 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
W20260929 19:00:44.194307 13582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:00:44.194839 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:44.183724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.98.62:35785)
0929 19:00:44.183829 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:44.183831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:44.183842 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:44.183868 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:44.183868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:44.183880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:44.183973 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:44.184667 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.185394 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.186423 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:44.186757 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:44.186835 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:44.189831 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:44.189832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:44.189833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:44.189839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:44.192889 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.193651 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.193652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.193653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.193719 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:44.194151 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:44.194152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:44.194152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:44.194289 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:44.194465 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:44.194521 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 19:00:44.194638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
Sep 29 19:00:44 dist-test-slave-gfsd krb5kdc[12120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:44.798226 13314 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:44.798395 13314 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:44.798846 13116 ts_manager.cc:194] Registered new tserver with Master: cc0c9ea314994493ac53ded39c633d52 (127.0.98.1:41341)
I20260929 19:00:44.799922 13116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47455
I20260929 19:00:45.011556 13448 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:45.011711 13448 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:45.012070 13115 ts_manager.cc:194] Registered new tserver with Master: b9e9af98e0d34bfc89fb6da183e0e322 (127.0.98.2:46577)
I20260929 19:00:45.012645 13115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32941
Sep 29 19:00:45 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:45.194504 13582 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:45.194686 13582 heartbeater.cc:511] Master 127.0.98.62:35785 requested a full tablet report, sending...
I20260929 19:00:45.195047 13116 ts_manager.cc:194] Registered new tserver with Master: 88d460adc30d484680db88fe312afb7c (127.0.98.3:40331)
I20260929 19:00:45.195600 13116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50209
I20260929 19:00:45.201898 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:45.211315 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.202908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.0.98.62:35785)
0929 19:00:45.203043 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.203045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.203213 (+ 168us) server_negotiation.cc:407] Connection header received
0929 19:00:45.203252 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.203253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.203267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.203373 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:45.204124 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.204948 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.206021 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.206348 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.206430 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.208024 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:45.208026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:45.208026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:45.208034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:45.209895 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:45.210338 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:45.210339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:45.210340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:45.210409 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:45.210658 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:45.210659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:45.210660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:45.210808 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:45.210909 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.211075 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.211164 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260929 19:00:45.214005 13116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:45.217613 13116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:45.226176 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.220465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41341 (local address 127.0.0.1:37664)
0929 19:00:45.221747 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.221756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:45.221808 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:45.222099 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:45.222102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:45.222104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:45.222179 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.222182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.223499 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.223499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.224806 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.224809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.225856 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.225857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.225938 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.225954 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 19:00:45.226020 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:00:45.226074 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260929 19:00:45.226493 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.220944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.1:41341)
0929 19:00:45.221055 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.221060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.221862 (+ 802us) server_negotiation.cc:407] Connection header received
0929 19:00:45.221903 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.221906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.221924 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.221997 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:45.222267 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.223406 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.224919 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.225671 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.225730 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.225795 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.226368 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.226417 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":59,"threads_started":1}
I20260929 19:00:45.226532 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.220720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46577 (local address 127.0.0.1:52106)
0929 19:00:45.221898 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.221907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:45.221944 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:45.222208 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:45.222208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:45.222210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:45.222248 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.222249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.223920 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.223921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.225039 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.225042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.226333 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.226334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.226391 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.226396 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:45.226424 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 19:00:45.226462 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":667,"threads_started":1}
I20260929 19:00:45.226943 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.220859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.0.98.2:46577)
0929 19:00:45.222052 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.222057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.222065 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:45.222097 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.222099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.222113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.222319 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:45.222387 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.223828 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.225148 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.226232 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.226697 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.226781 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.226808 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.226847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":56,"threads_started":1}
I20260929 19:00:45.227236 13248 tablet_service.cc:1467] Processing CreateTablet for tablet 108892abea1e469fa2a9daf453d3c118 (DEFAULT_TABLE table=test-table [id=9e26f32b7a7248e4ab6fbf2b79ae8b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:45.227672 13248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108892abea1e469fa2a9daf453d3c118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:45.227630 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 108892abea1e469fa2a9daf453d3c118 (DEFAULT_TABLE table=test-table [id=9e26f32b7a7248e4ab6fbf2b79ae8b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:45.227962 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108892abea1e469fa2a9daf453d3c118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:45.230479 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.220465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40331 (local address 127.0.0.1:43360)
0929 19:00:45.221441 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.221483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:45.221586 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:45.226815 (+ 5229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:45.226815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:45.226817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:45.226855 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.226857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.228131 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.228132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.229216 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.229219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.230244 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.230245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.230307 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.230311 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:45.230360 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:00:45.230401 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":70,"threads_started":1}
I20260929 19:00:45.230746 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.221144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.3:40331)
0929 19:00:45.221264 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.221268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.221546 (+ 278us) server_negotiation.cc:407] Connection header received
0929 19:00:45.221697 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.221700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.221719 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.221779 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:45.226944 (+ 5165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.228005 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.229326 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.230039 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.230102 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.230179 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.230628 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.230670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260929 19:00:45.231127 13616 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Bootstrap starting.
I20260929 19:00:45.231374 13504 tablet_service.cc:1467] Processing CreateTablet for tablet 108892abea1e469fa2a9daf453d3c118 (DEFAULT_TABLE table=test-table [id=9e26f32b7a7248e4ab6fbf2b79ae8b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:45.231678 13504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108892abea1e469fa2a9daf453d3c118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:45.231791 13616 tablet_bootstrap.cc:654] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:45.232216 13616 log.cc:824] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:45.232995 13617 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Bootstrap starting.
I20260929 19:00:45.233534 13617 tablet_bootstrap.cc:654] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:45.233961 13617 log.cc:824] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:45.234723 13617 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: No bootstrap required, opened a new log
I20260929 19:00:45.234790 13617 ts_tablet_manager.cc:1397] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:45.234917 13620 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Bootstrap starting.
I20260929 19:00:45.236101 13620 tablet_bootstrap.cc:654] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:45.236478 13620 log.cc:824] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:45.237720 13617 raft_consensus.cc:348] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.237883 13617 raft_consensus.cc:374] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:45.237924 13617 raft_consensus.cc:729] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e9af98e0d34bfc89fb6da183e0e322, State: Initialized, Role: FOLLOWER
I20260929 19:00:45.238075 13617 consensus_queue.cc:260] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.238340 13617 ts_tablet_manager.cc:1428] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260929 19:00:45.238488 13448 heartbeater.cc:503] Master 127.0.98.62:35785 was elected leader, sending a full tablet report...
I20260929 19:00:45.238930 13616 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: No bootstrap required, opened a new log
I20260929 19:00:45.238996 13616 ts_tablet_manager.cc:1397] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 19:00:45.239953 13620 tablet_bootstrap.cc:492] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: No bootstrap required, opened a new log
I20260929 19:00:45.240026 13620 ts_tablet_manager.cc:1397] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 19:00:45.242261 13616 raft_consensus.cc:348] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.242421 13616 raft_consensus.cc:374] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:45.242455 13616 raft_consensus.cc:729] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0c9ea314994493ac53ded39c633d52, State: Initialized, Role: FOLLOWER
I20260929 19:00:45.242602 13616 consensus_queue.cc:260] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.242545 13620 raft_consensus.cc:348] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.242674 13620 raft_consensus.cc:374] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:45.242709 13620 raft_consensus.cc:729] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d460adc30d484680db88fe312afb7c, State: Initialized, Role: FOLLOWER
I20260929 19:00:45.242862 13616 ts_tablet_manager.cc:1428] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 19:00:45.242969 13314 heartbeater.cc:503] Master 127.0.98.62:35785 was elected leader, sending a full tablet report...
I20260929 19:00:45.242987 13620 consensus_queue.cc:260] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.243513 13620 ts_tablet_manager.cc:1428] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:00:45.243669 13582 heartbeater.cc:503] Master 127.0.98.62:35785 was elected leader, sending a full tablet report...
W20260929 19:00:45.245252 13449 tablet.cc:2369] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:45.282927 13315 tablet.cc:2369] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:45.366333 13623 raft_consensus.cc:482] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:45.366506 13623 raft_consensus.cc:504] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.366982 13623 leader_election.cc:288] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9e9af98e0d34bfc89fb6da183e0e322 (127.0.98.2:46577), 88d460adc30d484680db88fe312afb7c (127.0.98.3:40331)
I20260929 19:00:45.371817 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.367406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56679 (local address 127.0.98.2:46577)
0929 19:00:45.367457 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.367459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.368522 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 19:00:45.368659 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.368660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.368680 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.368715 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:45.368991 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.369749 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.370909 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.371428 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.371519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.371531 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.371696 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.371740 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:00:45.371824 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.367256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46577 (local address 127.0.98.1:56679)
0929 19:00:45.368455 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.368467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:45.368554 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:45.368792 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:45.368792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:45.368794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:45.368888 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.368890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.369905 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.369906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.370802 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.370804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.371520 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.371521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.371602 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.371608 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:00:45.371674 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:00:45.371717 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":745,"threads_started":1}
I20260929 19:00:45.372555 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.367814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44961 (local address 127.0.98.3:40331)
0929 19:00:45.367860 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.367862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.369051 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 19:00:45.369157 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.369158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.369171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.369205 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:45.369407 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.370076 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.371452 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.372108 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.372191 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.372201 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.372430 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.372473 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:00:45.372485 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108892abea1e469fa2a9daf453d3c118" candidate_uuid: "cc0c9ea314994493ac53ded39c633d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" is_pre_election: true
I20260929 19:00:45.372689 13402 raft_consensus.cc:2454] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0c9ea314994493ac53ded39c633d52 in term 0.
I20260929 19:00:45.372808 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.367613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40331 (local address 127.0.98.1:44961)
0929 19:00:45.368995 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.369004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:45.369077 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:45.369268 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:45.369269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:45.369270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:45.369328 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.369330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.370200 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.370201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.371353 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:45.371356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.372231 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.372233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:45.372328 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.372333 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:45.372666 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 19:00:45.372716 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"thread_start_us":91,"threads_started":1}
I20260929 19:00:45.373095 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108892abea1e469fa2a9daf453d3c118" candidate_uuid: "cc0c9ea314994493ac53ded39c633d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d460adc30d484680db88fe312afb7c" is_pre_election: true
I20260929 19:00:45.373227 13201 leader_election.cc:302] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e9af98e0d34bfc89fb6da183e0e322, cc0c9ea314994493ac53ded39c633d52; no voters:
I20260929 19:00:45.373260 13533 raft_consensus.cc:2454] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0c9ea314994493ac53ded39c633d52 in term 0.
I20260929 19:00:45.373359 13623 raft_consensus.cc:2781] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:45.373422 13623 raft_consensus.cc:482] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:45.373450 13623 raft_consensus.cc:3037] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:45.374465 13623 raft_consensus.cc:504] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.374639 13623 leader_election.cc:288] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [CANDIDATE]: Term 1 election: Requested vote from peers b9e9af98e0d34bfc89fb6da183e0e322 (127.0.98.2:46577), 88d460adc30d484680db88fe312afb7c (127.0.98.3:40331)
I20260929 19:00:45.374830 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108892abea1e469fa2a9daf453d3c118" candidate_uuid: "cc0c9ea314994493ac53ded39c633d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d460adc30d484680db88fe312afb7c"
I20260929 19:00:45.374922 13533 raft_consensus.cc:3037] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:45.375345 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108892abea1e469fa2a9daf453d3c118" candidate_uuid: "cc0c9ea314994493ac53ded39c633d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e9af98e0d34bfc89fb6da183e0e322"
I20260929 19:00:45.375429 13402 raft_consensus.cc:3037] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:45.375945 13533 raft_consensus.cc:2454] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0c9ea314994493ac53ded39c633d52 in term 1.
I20260929 19:00:45.376305 13402 raft_consensus.cc:2454] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0c9ea314994493ac53ded39c633d52 in term 1.
I20260929 19:00:45.376505 13201 leader_election.cc:302] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e9af98e0d34bfc89fb6da183e0e322, cc0c9ea314994493ac53ded39c633d52; no voters:
I20260929 19:00:45.376725 13623 raft_consensus.cc:2781] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:45.376947 13623 raft_consensus.cc:686] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [term 1 LEADER]: Becoming Leader. State: Replica: cc0c9ea314994493ac53ded39c633d52, State: Running, Role: LEADER
I20260929 19:00:45.377054 13623 consensus_queue.cc:237] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } }
I20260929 19:00:45.377942 13115 catalog_manager.cc:5495] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc0c9ea314994493ac53ded39c633d52 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc0c9ea314994493ac53ded39c633d52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0c9ea314994493ac53ded39c633d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 } health_report { overall_health: UNKNOWN } } }
Sep 29 19:00:45 dist-test-slave-gfsd krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:00:45.410465 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.406956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.98.1:41341)
0929 19:00:45.407057 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:45.407059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:45.407068 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:45.407111 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:45.407112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:45.407139 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:45.407219 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:45.407434 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.408098 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.409289 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:45.409557 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:45.409626 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:45.409869 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:45.410010 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:45.410127 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:45.410364 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 19:00:45.410388 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 19:00:45.412596 13533 raft_consensus.cc:1261] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 1 FOLLOWER]: Refusing update from remote peer cc0c9ea314994493ac53ded39c633d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:45.412647 13402 raft_consensus.cc:1261] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 1 FOLLOWER]: Refusing update from remote peer cc0c9ea314994493ac53ded39c633d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:45.412935 13623 consensus_queue.cc:1035] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e9af98e0d34bfc89fb6da183e0e322" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:45.413050 13628 consensus_queue.cc:1035] T 108892abea1e469fa2a9daf453d3c118 P cc0c9ea314994493ac53ded39c633d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d460adc30d484680db88fe312afb7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:45.416244 13631 mvcc.cc:205] Tried to move back new op lower bound from 7334741792407318528 to 7334741792265048064. Current Snapshot: MvccSnapshot[applied={T|T < 7334741792407318528}]
I20260929 19:00:45.416626 13633 mvcc.cc:205] Tried to move back new op lower bound from 7334741792407318528 to 7334741792265048064. Current Snapshot: MvccSnapshot[applied={T|T < 7334741792407318528}]
I20260929 19:00:45.417989 13632 mvcc.cc:205] Tried to move back new op lower bound from 7334741792407318528 to 7334741792265048064. Current Snapshot: MvccSnapshot[applied={T|T < 7334741792407318528}]
I20260929 19:00:45.420254 13115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:45.420351 13115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:45.421027 13115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82ff0a43b783498595cdfacf7872706b: Sending DeleteTablet for 3 replicas of tablet 108892abea1e469fa2a9daf453d3c118
I20260929 19:00:45.421443 13248 tablet_service.cc:1514] Processing DeleteTablet for tablet 108892abea1e469fa2a9daf453d3c118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37664
I20260929 19:00:45.421494 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 108892abea1e469fa2a9daf453d3c118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52106
I20260929 19:00:45.421686 13504 tablet_service.cc:1514] Processing DeleteTablet for tablet 108892abea1e469fa2a9daf453d3c118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43360
I20260929 19:00:45.421877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13154
I20260929 19:00:45.422540 13641 tablet_replica.cc:330] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: stopping tablet replica
I20260929 19:00:45.422709 13641 raft_consensus.cc:2229] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:45.422891 13641 raft_consensus.cc:2258] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:45.423418 13641 ts_tablet_manager.cc:1905] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:45.424646 13642 tablet_replica.cc:330] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: stopping tablet replica
I20260929 19:00:45.424773 13642 raft_consensus.cc:2229] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:45.424872 13642 raft_consensus.cc:2258] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:45.425026 13641 ts_tablet_manager.cc:1918] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:45.425094 13641 log.cc:1196] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/108892abea1e469fa2a9daf453d3c118
I20260929 19:00:45.425344 13641 ts_tablet_manager.cc:1939] T 108892abea1e469fa2a9daf453d3c118 P b9e9af98e0d34bfc89fb6da183e0e322: Deleting consensus metadata
I20260929 19:00:45.425758 13100 catalog_manager.cc:4841] TS b9e9af98e0d34bfc89fb6da183e0e322 (127.0.98.2:46577): tablet 108892abea1e469fa2a9daf453d3c118 (table test-table [id=9e26f32b7a7248e4ab6fbf2b79ae8b51]) successfully deleted
I20260929 19:00:45.426102 13642 ts_tablet_manager.cc:1905] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:45.427603 13642 ts_tablet_manager.cc:1918] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:45.427686 13642 log.cc:1196] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/108892abea1e469fa2a9daf453d3c118
I20260929 19:00:45.427959 13642 ts_tablet_manager.cc:1939] T 108892abea1e469fa2a9daf453d3c118 P 88d460adc30d484680db88fe312afb7c: Deleting consensus metadata
I20260929 19:00:45.428464 13101 catalog_manager.cc:4841] TS 88d460adc30d484680db88fe312afb7c (127.0.98.3:40331): tablet 108892abea1e469fa2a9daf453d3c118 (table test-table [id=9e26f32b7a7248e4ab6fbf2b79ae8b51]) successfully deleted
W20260929 19:00:45.428800 13100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41341 (error 108)
I20260929 19:00:45.429376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13318
I20260929 19:00:45.436544 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:45.436411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41341 (local address 127.0.0.1:37696)
0929 19:00:45.436468 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:45.436505 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 19:00:45.436698 13100 catalog_manager.cc:4584] TS cc0c9ea314994493ac53ded39c633d52 (127.0.98.1:41341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 108892abea1e469fa2a9daf453d3c118: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41341: connect: Connection refused (error 111)
I20260929 19:00:45.437263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13450
I20260929 19:00:45.442817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13083
I20260929 19:00:45.450258 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 19:00:45.805061 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 19:00:45.805177 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 19:00:46.149765 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 19:00:46.149 UTC [12150] LOG: received smart shutdown request
2026-09-29 19:00:46.153 UTC [12150] LOG: background worker "logical replication launcher" (PID 12159) exited with exit code 1
2026-09-29 19:00:46.154 UTC [12154] LOG: shutting down
2026-09-29 19:00:46.324 UTC [12150] LOG: database system is shut down
2026-09-29T19:00:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47428 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:46 dist-test-slave-gfsd krb5kdc[13653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:46 dist-test-slave-gfsd krb5kdc[13653](info): set up 2 sockets
Sep 29 19:00:46 dist-test-slave-gfsd krb5kdc[13653](info): commencing operation
krb5kdc: starting...
W20260929 19:00:47.535190 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:47.654626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:47.734995 13666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:47.735237 13666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:47.735273 13666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:47.736398 13666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:47.736465 13666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:47.736491 13666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:47.736522 13666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:47.736549 13666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:47.738057 13666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:47.738324 13666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:47.738543 13666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:47.743381 13672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:47.743494 13666 server_base.cc:1034] running on GCE node
W20260929 19:00:47.743450 13671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:47.743474 13674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:47.743845 13666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:47.744138 13666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:47.745306 13666 hybrid_clock.cc:648] HybridClock initialized: now 1790708447745285 us; error 37 us; skew 500 ppm
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:47.747767 13666 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:47.748236 13666 webserver.cc:460] Webserver started at http://127.0.98.62:38787/ using document root <none> and password file <none>
I20260929 19:00:47.748461 13666 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:47.748528 13666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:47.748678 13666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:47.749691 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "ab9aabb4ff094bc18e3f92de4a816586"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "233f18d5eb5e050534ba50c77dfc9513"
server_key_iv: "e24a0c27a51b23ac8d6c09f3399da69b"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.750061 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab9aabb4ff094bc18e3f92de4a816586"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "233f18d5eb5e050534ba50c77dfc9513"
server_key_iv: "e24a0c27a51b23ac8d6c09f3399da69b"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.751410 13666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:47.752194 13681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:47.752373 13666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:47.752476 13666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "ab9aabb4ff094bc18e3f92de4a816586"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "233f18d5eb5e050534ba50c77dfc9513"
server_key_iv: "e24a0c27a51b23ac8d6c09f3399da69b"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.752568 13666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:47.760844 13666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:47.763084 13666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:47.763239 13666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:47.767091 13666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42511
I20260929 19:00:47.767122 13733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42511 every 8 connection(s)
I20260929 19:00:47.767484 13666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:47.768538 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:47.769351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13666
I20260929 19:00:47.769455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:47.769657 392 external_mini_cluster.cc:1428] Setting key 091532ffc1742f2f1e907aed57d6bf39
I20260929 19:00:47.770759 13734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 19:00:47.782083 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.771655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.0.98.62:42511)
0929 19:00:47.771796 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:47.771806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:47.771942 (+ 136us) server_negotiation.cc:407] Connection header received
0929 19:00:47.772733 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:47.772735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:47.772753 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:47.772837 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:47.773620 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.774434 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.775881 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.776190 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.776273 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.778635 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:47.778642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:47.778643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:47.778650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:47.780348 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.780785 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.780788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.780789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.780900 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.781278 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.781279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.781279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.781487 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:47.781560 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:47.781813 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 19:00:47.781918 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260929 19:00:47.782320 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.771596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.0.1:48828)
0929 19:00:47.771792 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.771839 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:47.772584 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:47.772873 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:47.772875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:47.773280 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:47.773505 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.773513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.774534 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.774536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.775770 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.775772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.776281 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.776283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.776373 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.776889 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:47.776899 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:47.778513 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:47.780427 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.780433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.780435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.780668 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.780988 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.780989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.780989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.781111 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.781567 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:47.781574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:47.781653 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 19:00:47.781760 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:00:47.782021 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":85,"thread_start_us":80,"threads_started":1}
I20260929 19:00:47.783141 13734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:00:47.783536 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.783415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55060)
0929 19:00:47.783465 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.783508 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 19:00:47.783655 13682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
I20260929 19:00:47.784072 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.783980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55074)
0929 19:00:47.784026 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.784051 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:00:47.784175 13734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:00:47.798230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42443
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:47.833432 13734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } attempt: 1
I20260929 19:00:47.833947 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.833816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55088)
0929 19:00:47.833878 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.833919 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:00:47.834369 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.834271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55094)
0929 19:00:47.834323 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.834354 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 19:00:47.834460 13734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
W20260929 19:00:47.879812 13741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:47.880015 13741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:47.880038 13741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:47.881173 13741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:47.881213 13741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:47.881228 13741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:47.881279 13741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:47.881297 13741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:47.882810 13741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:47.883064 13741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:47.883311 13741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:47.888005 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:47.888167 13741 server_base.cc:1034] running on GCE node
W20260929 19:00:47.888093 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:47.888000 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:47.888582 13741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:47.888901 13741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:47.890058 13741 hybrid_clock.cc:648] HybridClock initialized: now 1790708447890030 us; error 44 us; skew 500 ppm
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:47.892735 13741 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:00:47.893214 13741 webserver.cc:460] Webserver started at http://127.0.98.61:44741/ using document root <none> and password file <none>
I20260929 19:00:47.893441 13741 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:47.893508 13741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:47.893836 13741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:47.895026 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data/instance:
uuid: "b41a4630dc014a88ba05b0e175469b7e"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "62e4aae87869a1920d5b263aed968cad"
server_key_iv: "46478dd42bf040075956a66dff33e961"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.895473 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal/instance:
uuid: "b41a4630dc014a88ba05b0e175469b7e"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "62e4aae87869a1920d5b263aed968cad"
server_key_iv: "46478dd42bf040075956a66dff33e961"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.896798 13741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:47.897570 13756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:47.897745 13741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:47.897845 13741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "b41a4630dc014a88ba05b0e175469b7e"
format_stamp: "Formatted at 2026-09-29 19:00:47 on dist-test-slave-gfsd"
server_key: "62e4aae87869a1920d5b263aed968cad"
server_key_iv: "46478dd42bf040075956a66dff33e961"
server_key_version: "encryptionkey@0"
I20260929 19:00:47.897928 13741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:47.902634 13734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } attempt: 2
I20260929 19:00:47.903038 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.902916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55098)
0929 19:00:47.902974 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.903012 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:00:47.903462 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.903340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55102)
0929 19:00:47.903388 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.903418 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:00:47.903564 13734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42443: connect: Connection refused (error 111)
I20260929 19:00:47.906113 13741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:47.908034 13741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:47.908198 13741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:47.912177 13741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42443
I20260929 19:00:47.912200 13808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42443 every 8 connection(s)
I20260929 19:00:47.912544 13741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:00:47.912935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13741
I20260929 19:00:47.913039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-1/wal/instance
I20260929 19:00:47.913239 392 external_mini_cluster.cc:1428] Setting key 48ce80c252438bb827710c10c7bca687
I20260929 19:00:47.913507 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:47.915438 13809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:47.928205 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.916333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.98.62:42511)
0929 19:00:47.916399 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:47.916402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:47.916620 (+ 218us) server_negotiation.cc:407] Connection header received
0929 19:00:47.917136 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:47.917138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:47.917151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:47.917230 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:47.917860 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.918701 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.919818 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.920168 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.920249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.924657 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:47.924659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:47.924660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:47.924668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:47.926418 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.926977 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.926979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.926979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.927051 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.927459 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.927460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.927461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.927622 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:47.927713 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:47.928005 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 19:00:47.928103 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:00:47.928407 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.916301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.0.1:48844)
0929 19:00:47.916485 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.916527 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:47.917033 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:47.917297 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:47.917300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:47.917573 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:47.917766 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.917771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.918812 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.918813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.919728 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.919731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.920261 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.920262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.920361 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.920933 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:47.920969 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:47.924559 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:47.926520 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.926527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.926529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.926846 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.927202 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.927203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.927203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.927383 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.927703 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:47.927711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:47.927862 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 19:00:47.927971 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:00:47.928161 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260929 19:00:47.929129 13809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:00:47.937837 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.929411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55104)
0929 19:00:47.929468 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.929479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:47.929528 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:47.929791 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:47.929792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:47.929939 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:47.929984 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.929985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.930924 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.930925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.931503 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.931505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.931978 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.931979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.932048 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.932481 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:47.932485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:47.934542 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:47.936572 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.936573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.936573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.936743 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.937048 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:47.937051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:47.937052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:47.937125 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:47.937448 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:47.937450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:47.937506 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:00:47.937654 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 19:00:47.937765 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:00:47.938015 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.929491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.0.98.61:42443)
0929 19:00:47.929656 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:47.929660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:47.929681 (+ 21us) server_negotiation.cc:407] Connection header received
0929 19:00:47.929697 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:47.929699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:47.929710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:47.929791 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:47.930056 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.930824 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.931591 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.931896 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.931983 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.934642 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:47.934650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:47.934651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:47.934666 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:47.936478 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.936840 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.936843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.936843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.936949 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:47.937203 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:47.937204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:47.937204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:47.937359 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:47.937432 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:47.937686 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 19:00:47.937835 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260929 19:00:47.938740 13809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:00:47.939121 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.938999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47218)
0929 19:00:47.939048 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.939091 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:00:47.939335 13758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260929 19:00:47.939760 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.939669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47228)
0929 19:00:47.939716 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.939740 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:00:47.939865 13809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:00:47.941905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42257
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:47.989089 13809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 1
I20260929 19:00:47.989792 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.989625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47236)
0929 19:00:47.989714 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.989761 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 19:00:47.990166 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.990064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47246)
0929 19:00:47.990111 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.990146 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:00:47.990254 13809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260929 19:00:47.994721 13734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } attempt: 3
Sep 29 19:00:47 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:00:48.002468 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.995056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55120)
0929 19:00:47.995109 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:47.995194 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:47.995277 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:47.995459 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:47.995460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:47.995625 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:47.995687 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.995689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.996459 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.996460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.996963 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:47.996965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.997376 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.997377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:47.997461 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.998020 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:47.998025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:47.999325 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.001207 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.001208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.001208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.001429 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.001731 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.001732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.001732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.001800 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.002101 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.002102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.002158 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.002250 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.002361 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:00:48.002466 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:47.995126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.98.61:42443)
0929 19:00:47.995193 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:47.995195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:47.995258 (+ 63us) server_negotiation.cc:407] Connection header received
0929 19:00:47.995371 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:47.995372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:47.995383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:47.995452 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:47.995778 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.996376 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.997063 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:47.997283 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:47.997375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:47.999417 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:47.999418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:47.999419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:47.999427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.000993 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.001580 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.001581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.001582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.001644 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.001883 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.001883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.001884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.002020 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.002113 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.002275 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.002362 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:00:48.002964 13734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:00:48.003434 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.003267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47256)
0929 19:00:48.003365 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.003402 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 19:00:48.003895 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.003800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47258)
0929 19:00:48.003848 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.003873 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:00:48.003993 13734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
W20260929 19:00:48.025645 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:48.025877 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:48.025899 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:48.027066 13816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:48.027115 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:48.027130 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:48.027143 13816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:48.027155 13816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:48.028790 13816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511,127.0.98.61:42443,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:48.029057 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:48.029332 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:48.034273 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:48.034283 13822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:48.034329 13821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:48.034569 13816 server_base.cc:1034] running on GCE node
I20260929 19:00:48.034804 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:48.035045 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:48.035151 13734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 1
I20260929 19:00:48.035569 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.035439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47274)
0929 19:00:48.035510 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.035542 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 19:00:48.035930 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.035832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47276)
0929 19:00:48.035889 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.035914 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 19:00:48.036015 13734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260929 19:00:48.036186 13816 hybrid_clock.cc:648] HybridClock initialized: now 1790708448036154 us; error 50 us; skew 500 ppm
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:48.038729 13816 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:00:48.039132 13816 webserver.cc:460] Webserver started at http://127.0.98.60:35851/ using document root <none> and password file <none>
I20260929 19:00:48.039453 13816 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:48.039508 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:48.039606 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:48.040701 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data/instance:
uuid: "4a267b86d1f64cc3a09a547a62c691df"
format_stamp: "Formatted at 2026-09-29 19:00:48 on dist-test-slave-gfsd"
server_key: "bc504c67a6f88141b726506c9991e121"
server_key_iv: "24c595c749cd43e4af84b4e33bcb04de"
server_key_version: "encryptionkey@0"
I20260929 19:00:48.041021 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal/instance:
uuid: "4a267b86d1f64cc3a09a547a62c691df"
format_stamp: "Formatted at 2026-09-29 19:00:48 on dist-test-slave-gfsd"
server_key: "bc504c67a6f88141b726506c9991e121"
server_key_iv: "24c595c749cd43e4af84b4e33bcb04de"
server_key_version: "encryptionkey@0"
I20260929 19:00:48.042424 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:00:48.043133 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:48.043381 13816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:48.043458 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "4a267b86d1f64cc3a09a547a62c691df"
format_stamp: "Formatted at 2026-09-29 19:00:48 on dist-test-slave-gfsd"
server_key: "bc504c67a6f88141b726506c9991e121"
server_key_iv: "24c595c749cd43e4af84b4e33bcb04de"
server_key_version: "encryptionkey@0"
I20260929 19:00:48.043511 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:48.058398 13809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 2
I20260929 19:00:48.058857 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.058704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47278)
0929 19:00:48.058771 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.058818 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:00:48.059255 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.059141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47294)
0929 19:00:48.059201 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.059227 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:00:48.059391 13809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260929 19:00:48.060506 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:48.062865 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:48.063048 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:48.067036 13816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42257
I20260929 19:00:48.067073 13883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42257 every 8 connection(s)
I20260929 19:00:48.067497 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:00:48.068363 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:48.069968 13884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:48.077323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13816
I20260929 19:00:48.077446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708293967505-392-0/minicluster-data/master-2/wal/instance
I20260929 19:00:48.077733 392 external_mini_cluster.cc:1428] Setting key 967a664d8cd2ab6b9d0c7a46b3bbcb0b
I20260929 19:00:48.080499 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.071014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.0.98.62:42511)
0929 19:00:48.071072 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.071074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.071372 (+ 298us) server_negotiation.cc:407] Connection header received
0929 19:00:48.071859 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.071861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.071873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.071947 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.072660 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.073203 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.074250 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.074451 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.074513 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.076963 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.076965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.076965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.076973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.078941 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.079454 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.079456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.079456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.079525 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.079827 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.079828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.079828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.079939 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.079990 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.080354 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.080423 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:00:48.080811 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.070936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.0.1:48846)
0929 19:00:48.071203 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.071238 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:48.071755 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:48.072016 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:48.072020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:48.072358 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:48.072568 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.072573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.073308 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.073309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.074106 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.074113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.074553 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.074554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.074693 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.075372 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:48.075386 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:48.076862 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.079035 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.079041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.079043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.079381 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.079606 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.079607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.079607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.079743 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.080003 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.080009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.080174 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.080279 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.080542 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":85,"thread_start_us":110,"threads_started":1}
I20260929 19:00:48.081373 13884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:00:48.088271 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.078409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.60:42257)
0929 19:00:48.078580 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.078585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.078612 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:00:48.078633 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.078634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.078645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.078737 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.079508 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.080611 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.081607 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.081969 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.082052 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.084514 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.084522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.084522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.084528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.086822 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.087307 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.087311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.087311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.087377 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.087662 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.087662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.087663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.087864 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.088020 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.088068 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.088141 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260929 19:00:48.089574 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.081694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42443 (local address 127.0.0.1:55128)
0929 19:00:48.081752 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.081761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:48.081798 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:48.081963 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:48.081964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:48.082098 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:48.082205 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.082207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.082830 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.082831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.083446 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.083448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.083828 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.083829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.083900 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.084286 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:48.084290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:48.085898 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.088541 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.088542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.088542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.088713 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.088967 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.088968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.088968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.089032 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.089367 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.089368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.089402 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.089451 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.089515 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 19:00:48.089663 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.081770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.0.98.61:42443)
0929 19:00:48.081836 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.081838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.081845 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:48.081871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.081872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.081885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.081944 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.082289 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.082740 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.083536 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.083728 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.083830 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.085995 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.085997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.085997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.086005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.088463 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.088835 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.088836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.088836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.088895 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.089159 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.089160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.089160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.089297 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.089375 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.089481 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.089557 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:00:48.090034 13884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:00:48.096992 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.090281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47304)
0929 19:00:48.090330 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.090342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:48.090390 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:48.090507 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:48.090508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:48.090639 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:48.090662 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.090663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.091372 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.091373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.091887 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.091890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.092496 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.092497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.092537 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.092912 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:48.092916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:48.094319 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.095967 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.095968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.095968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.096180 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.096391 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.096392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.096392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.096445 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.096686 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.096687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.096723 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.096790 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.096877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:00:48.097095 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.090312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.0.98.60:42257)
0929 19:00:48.090342 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.090344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.090375 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:00:48.090442 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.090443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.090452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.090504 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.090765 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.091251 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.092006 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.092403 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.092481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.094412 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.094431 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.094431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.094437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.095895 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.096270 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.096271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.096271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.096319 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.096515 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.096516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.096516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.096620 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.096693 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.096816 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.096977 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260929 19:00:48.097427 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.089185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.0.98.62:42511)
0929 19:00:48.089233 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.089235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.089268 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:00:48.089297 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.089299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.089310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.089359 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.089974 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.090576 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.091311 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.091515 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.091605 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.093982 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.093983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.093983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.093990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.095706 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.096014 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.096015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.096015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.096066 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.096363 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.096365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.096365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.096490 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.097215 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.097266 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.097342 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:00:48.098618 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: Bootstrap starting.
I20260929 19:00:48.099603 13884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:48.099897 13884 log.cc:824] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:48.100756 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: No bootstrap required, opened a new log
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:00:48.102810 13884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.103003 13884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:48.103045 13884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a267b86d1f64cc3a09a547a62c691df, State: Initialized, Role: FOLLOWER
I20260929 19:00:48.103201 13884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.103550 13890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.103628 13884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:48.103649 13890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:00:48.105789 13901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 19:00:48.105943 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.098077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.98.61:42443)
0929 19:00:48.098119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.098120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.098128 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:48.098151 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.098153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.098164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.098217 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.098639 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.099364 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.100155 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.100367 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.100425 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.102894 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.102896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.102896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.102903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.104770 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.105120 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.105121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.105122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.105189 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.105399 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.105400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.105401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.105529 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.105719 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.105759 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.105832 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:00:48.105970 13901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:00:48.107383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:00:48.106024 13816 master_runner.cc:186] Error getting master registration for 127.0.98.62:42511: OK, instance_id { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" instance_seqno: 1790708447765965 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:00:48.108453 13816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:00:48.111125 13734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 2
I20260929 19:00:48.117760 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.109036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.98.61:42443)
0929 19:00:48.109071 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.109073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.109335 (+ 262us) server_negotiation.cc:407] Connection header received
0929 19:00:48.109428 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.109430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.109440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.109470 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.110206 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.110907 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.111999 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.112280 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.112319 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.114259 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.114260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.114260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.114268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.116059 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.116712 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.116713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.116713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.116769 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.116966 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.116966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.116967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.117093 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.117186 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.117567 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.117654 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:00:48.118065 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.108823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.98.62:42511)
0929 19:00:48.108863 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.108865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.109103 (+ 238us) server_negotiation.cc:407] Connection header received
0929 19:00:48.109366 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.109367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.109380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.109412 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.110179 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.110905 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.112411 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.112713 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.112753 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.114259 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.114260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.114260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.114267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.116810 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.117248 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.117249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.117249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.117302 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.117508 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.117509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.117510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.117632 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.117701 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.117905 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.117978 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:00:48.118202 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.108940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.98.60:42257)
0929 19:00:48.108975 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.108977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.109135 (+ 158us) server_negotiation.cc:407] Connection header received
0929 19:00:48.109405 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.109406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.109416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.109454 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.110241 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.110946 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.112098 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.112382 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.112423 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.114349 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.114350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.114350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.114357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.116800 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.117293 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.117294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.117295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.117351 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.117556 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.117557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.117558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.117739 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.117833 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.118052 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.118126 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:00:48.123092 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.111364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47310)
0929 19:00:48.111450 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.111460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:48.111493 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:48.112817 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:48.112818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:48.112990 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:48.113027 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.113029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.115504 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.115505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.116328 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.116330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.118306 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.118308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.118364 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.118891 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:48.118896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:48.120263 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.121928 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.121929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.121929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.122167 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.122410 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.122411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.122411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.122479 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.122753 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.122754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.122800 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.122896 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.122994 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 19:00:48.123399 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.111554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.0.98.60:42257)
0929 19:00:48.112628 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.112639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.112646 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:48.112665 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.112666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.112676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.114500 (+ 1824us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.114534 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.115407 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.117895 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.118211 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.119355 (+ 1144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.120351 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.120352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.120353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.120360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.121804 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.122257 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.122259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.122259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.122322 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.122558 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.122559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.122559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.122678 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.122761 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.122922 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.122996 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":87,"threads_started":1}
I20260929 19:00:48.124817 13734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: Bootstrap starting.
I20260929 19:00:48.125475 13734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:48.125787 13734 log.cc:824] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:48.126603 13734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: No bootstrap required, opened a new log
I20260929 19:00:48.128526 13734 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.128679 13734 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:48.128731 13734 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9aabb4ff094bc18e3f92de4a816586, State: Initialized, Role: FOLLOWER
I20260929 19:00:48.128885 13734 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.129199 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.129261 13734 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:48.129328 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:00:48.131320 13923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:00:48.131381 13923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:00:48.132144 13666 master_runner.cc:186] Error getting master registration for 127.0.98.61:42443: OK, instance_id { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" instance_seqno: 1790708447911033 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:00:48.134407 13666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:00:48.150555 13809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 3
Sep 29 19:00:48 dist-test-slave-gfsd krb5kdc[13653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:00:48.159436 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.150896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:47316)
0929 19:00:48.150993 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:48.151005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:48.151045 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:48.151340 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:48.151341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:48.151540 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:48.151598 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.151600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.152607 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.152609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.153471 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:48.153473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.154023 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.154024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:48.154091 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.154669 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:48.154675 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:48.156230 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:48.158229 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.158230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.158231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.158453 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.158669 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:48.158669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:48.158670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:48.158757 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:48.159070 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:48.159071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:48.159121 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 19:00:48.159200 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:00:48.159327 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 19:00:48.159473 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:48.151145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.0.98.60:42257)
0929 19:00:48.151192 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:48.151194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:48.151202 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:48.151226 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:48.151227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:48.151236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:48.151710 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:48.151747 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.152512 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.153574 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:48.153939 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:48.154011 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:48.156322 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:48.156323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:48.156323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:48.156331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:48.158139 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.158531 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.158532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.158532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.158596 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:48.158835 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:48.158836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:48.158836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:48.158988 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:48.159055 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:48.159247 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 19:00:48.159375 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:00:48.161319 13809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: Bootstrap starting.
I20260929 19:00:48.162060 13809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:48.162339 13809 log.cc:824] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:48.163223 13809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: No bootstrap required, opened a new log
I20260929 19:00:48.165172 13809 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.165314 13809 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:48.165349 13809 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41a4630dc014a88ba05b0e175469b7e, State: Initialized, Role: FOLLOWER
I20260929 19:00:48.165529 13809 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.165851 13809 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:48.165820 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.166000 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:00:48.167776 13937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:00:48.167856 13937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:00:48.169838 13741 master_runner.cc:263] No leader master found from master b41a4630dc014a88ba05b0e175469b7e
I20260929 19:00:48.169883 13741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:00:48.186816 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:48.186966 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.187220 13912 leader_election.cc:288] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b41a4630dc014a88ba05b0e175469b7e (127.0.98.61:42443), 4a267b86d1f64cc3a09a547a62c691df (127.0.98.60:42257)
I20260929 19:00:48.187598 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab9aabb4ff094bc18e3f92de4a816586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41a4630dc014a88ba05b0e175469b7e" is_pre_election: true
I20260929 19:00:48.187774 13783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9aabb4ff094bc18e3f92de4a816586 in term 0.
I20260929 19:00:48.188052 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab9aabb4ff094bc18e3f92de4a816586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a267b86d1f64cc3a09a547a62c691df" is_pre_election: true
I20260929 19:00:48.188184 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9aabb4ff094bc18e3f92de4a816586 in term 0.
I20260929 19:00:48.188361 13683 leader_election.cc:302] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a267b86d1f64cc3a09a547a62c691df, ab9aabb4ff094bc18e3f92de4a816586; no voters:
I20260929 19:00:48.188719 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:48.188774 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:48.188797 13912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:48.189978 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.190095 13912 leader_election.cc:288] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [CANDIDATE]: Term 1 election: Requested vote from peers b41a4630dc014a88ba05b0e175469b7e (127.0.98.61:42443), 4a267b86d1f64cc3a09a547a62c691df (127.0.98.60:42257)
I20260929 19:00:48.190412 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab9aabb4ff094bc18e3f92de4a816586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a267b86d1f64cc3a09a547a62c691df"
I20260929 19:00:48.190490 13858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:48.191431 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab9aabb4ff094bc18e3f92de4a816586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41a4630dc014a88ba05b0e175469b7e"
I20260929 19:00:48.191505 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:48.192406 13783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9aabb4ff094bc18e3f92de4a816586 in term 1.
I20260929 19:00:48.192657 13682 leader_election.cc:302] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab9aabb4ff094bc18e3f92de4a816586, b41a4630dc014a88ba05b0e175469b7e; no voters:
I20260929 19:00:48.192824 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9aabb4ff094bc18e3f92de4a816586 in term 1.
I20260929 19:00:48.193073 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:48.193252 13912 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [term 1 LEADER]: Becoming Leader. State: Replica: ab9aabb4ff094bc18e3f92de4a816586, State: Running, Role: LEADER
I20260929 19:00:48.193346 13912 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260929 19:00:48.193759 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9aabb4ff094bc18e3f92de4a816586. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.193826 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:48.195650 13941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:48.195890 13941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:48.197441 13858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [term 1 FOLLOWER]: Refusing update from remote peer ab9aabb4ff094bc18e3f92de4a816586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:48.197834 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:48.199134 13890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9aabb4ff094bc18e3f92de4a816586. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.199218 13890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:00:48.199664 13783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [term 1 FOLLOWER]: Refusing update from remote peer ab9aabb4ff094bc18e3f92de4a816586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:48.199997 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:48.201294 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9aabb4ff094bc18e3f92de4a816586. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.201391 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:00:48.202361 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.202436 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:48.202858 13890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.202939 13890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:00:48.203166 13942 mvcc.cc:205] Tried to move back new op lower bound from 7334741803814363136 to 7334741803800449024. Current Snapshot: MvccSnapshot[applied={T|T < 7334741803814363136}]
I20260929 19:00:48.203811 13943 mvcc.cc:205] Tried to move back new op lower bound from 7334741803814363136 to 7334741803800449024. Current Snapshot: MvccSnapshot[applied={T|T < 7334741803814363136}]
I20260929 19:00:48.205441 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.205529 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:48.205753 13941 catalog_manager.cc:1347] Generated new cluster ID: 1826422e7dca424c8a6ce4f3894df776
I20260929 19:00:48.205801 13941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:00:48.205986 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9aabb4ff094bc18e3f92de4a816586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9aabb4ff094bc18e3f92de4a816586" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } peers { permanent_uuid: "b41a4630dc014a88ba05b0e175469b7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42443 } } peers { permanent_uuid: "4a267b86d1f64cc3a09a547a62c691df" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260929 19:00:48.206069 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:00:48.206318 13945 mvcc.cc:205] Tried to move back new op lower bound from 7334741803814363136 to 7334741803800449024. Current Snapshot: MvccSnapshot[applied={T|T < 7334741803814363136}]
I20260929 19:00:48.227777 13941 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:48.228556 13941 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:48.236917 13941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9aabb4ff094bc18e3f92de4a816586: Generated new TSK 0
I20260929 19:00:48.237144 13941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:00:49.106513 13901 catalog_manager.cc:1259] Loaded cluster ID: 1826422e7dca424c8a6ce4f3894df776
I20260929 19:00:49.106598 13901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: loading cluster ID for follower catalog manager: success
I20260929 19:00:49.107559 13901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: acquiring CA information for follower catalog manager: success
I20260929 19:00:49.107901 13901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a267b86d1f64cc3a09a547a62c691df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:00:49.168292 13937 catalog_manager.cc:1259] Loaded cluster ID: 1826422e7dca424c8a6ce4f3894df776
I20260929 19:00:49.168375 13937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: loading cluster ID for follower catalog manager: success
I20260929 19:00:49.169207 13937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: acquiring CA information for follower catalog manager: success
I20260929 19:00:49.169489 13937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b41a4630dc014a88ba05b0e175469b7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:00:49.288856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13666
I20260929 19:00:49.294071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13741
I20260929 19:00:49.298921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13816
2026-09-29T19:00:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2864 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:49 dist-test-slave-gfsd krb5kdc[13958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:49 dist-test-slave-gfsd krb5kdc[13958](info): set up 2 sockets
Sep 29 19:00:49 dist-test-slave-gfsd krb5kdc[13958](info): commencing operation
krb5kdc: starting...
W20260929 19:00:50.339476 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[13958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:50.458618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:50.538132 13971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:50.538334 13971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:50.538357 13971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:50.539448 13971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:50.539481 13971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:50.539495 13971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:50.539508 13971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:50.539520 13971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:50.540890 13971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:50.541075 13971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:50.541229 13971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:50.546015 13976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:50.546200 13971 server_base.cc:1034] running on GCE node
W20260929 19:00:50.546073 13977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:50.546084 13979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:50.546514 13971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:50.546813 13971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:50.547999 13971 hybrid_clock.cc:648] HybridClock initialized: now 1790708450547974 us; error 55 us; skew 500 ppm
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[13958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:50.550482 13971 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:50.550966 13971 webserver.cc:460] Webserver started at http://127.0.98.62:46059/ using document root <none> and password file <none>
I20260929 19:00:50.551200 13971 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:50.551272 13971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:50.551441 13971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:50.552508 13971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "78e3709ca3b144e4897b4dc928d76c69"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-gfsd"
server_key: "08068cf2d9fef1bcade9c4c2380c6187"
server_key_iv: "774dc82eb985fb18be86039ea0210fcf"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.552873 13971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "78e3709ca3b144e4897b4dc928d76c69"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-gfsd"
server_key: "08068cf2d9fef1bcade9c4c2380c6187"
server_key_iv: "774dc82eb985fb18be86039ea0210fcf"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.554154 13971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:50.554829 13986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:50.555001 13971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:50.555101 13971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "78e3709ca3b144e4897b4dc928d76c69"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-gfsd"
server_key: "08068cf2d9fef1bcade9c4c2380c6187"
server_key_iv: "774dc82eb985fb18be86039ea0210fcf"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.555184 13971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:50.566512 13971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:50.568863 13971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:50.569028 13971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:50.573577 13971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38623
I20260929 19:00:50.573611 14038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38623 every 8 connection(s)
I20260929 19:00:50.573962 13971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:50.574817 14039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:50.577191 14039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69: Bootstrap starting.
I20260929 19:00:50.577762 14039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:50.578055 14039 log.cc:824] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:50.578750 14039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69: No bootstrap required, opened a new log
I20260929 19:00:50.580402 14039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260929 19:00:50.580538 14039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:50.580560 14039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e3709ca3b144e4897b4dc928d76c69, State: Initialized, Role: FOLLOWER
I20260929 19:00:50.580713 14039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260929 19:00:50.580786 14039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:50.580829 14039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:50.580878 14039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:50.581411 14039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260929 19:00:50.581522 14039 leader_election.cc:302] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e3709ca3b144e4897b4dc928d76c69; no voters:
I20260929 19:00:50.581691 14039 leader_election.cc:288] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:50.581724 14042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:50.581943 14042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [term 1 LEADER]: Becoming Leader. State: Replica: 78e3709ca3b144e4897b4dc928d76c69, State: Running, Role: LEADER
I20260929 19:00:50.581955 14039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:50.582058 14042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260929 19:00:50.582499 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e3709ca3b144e4897b4dc928d76c69. Latest consensus state: current_term: 1 leader_uuid: "78e3709ca3b144e4897b4dc928d76c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } } }
I20260929 19:00:50.582645 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78e3709ca3b144e4897b4dc928d76c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e3709ca3b144e4897b4dc928d76c69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } } }
I20260929 19:00:50.582666 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:50.582772 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:50.582945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 13971
I20260929 19:00:50.583050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:50.583201 14051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:50.583240 392 external_mini_cluster.cc:1428] Setting key 222ca6d8f3d4db9687c3eee812264bad
I20260929 19:00:50.583789 14051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:50.585579 14051 catalog_manager.cc:1347] Generated new cluster ID: 66be4b2cbccf4910949bab6b500a4c25
I20260929 19:00:50.585649 14051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[13958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:50.596364 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:50.583927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.0.98.62:38623)
0929 19:00:50.584653 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:50.584661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:50.584687 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:00:50.584776 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:50.584781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:50.584825 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:50.585125 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:50.585883 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:50.587357 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:50.588453 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:50.589049 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:50.589135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:50.592327 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:50.592332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:50.592332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:50.592352 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:50.594579 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:50.595023 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:50.595026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:50.595026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:50.595129 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:50.595422 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:50.595423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:50.595424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:50.595721 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:50.595855 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:50.596005 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:00:50.596134 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":92,"threads_started":1}
I20260929 19:00:50.597491 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:00:50.597934 14051 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:50.598610 14051 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:50.602681 14051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e3709ca3b144e4897b4dc928d76c69: Generated new TSK 0
I20260929 19:00:50.602856 14051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:00:50.651782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 13971
2026-09-29T19:00:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1351 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[14066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[14066](info): set up 2 sockets
Sep 29 19:00:50 dist-test-slave-gfsd krb5kdc[14066](info): commencing operation
krb5kdc: starting...
W20260929 19:00:51.699457 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:51 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:51.818058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:51.898671 14079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:51.898867 14079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:51.898888 14079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:51.900060 14079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:51.900103 14079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:51.900117 14079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:51.900130 14079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:51.900142 14079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:51.901628 14079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:51.901867 14079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:51.902143 14079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:51.907162 14085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:51.907169 14087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:51.907390 14079 server_base.cc:1034] running on GCE node
W20260929 19:00:51.907264 14084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:51.907691 14079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:51.907934 14079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:51.909113 14079 hybrid_clock.cc:648] HybridClock initialized: now 1790708451909109 us; error 37 us; skew 500 ppm
Sep 29 19:00:51 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:51.915841 14079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:51.916318 14079 webserver.cc:460] Webserver started at http://127.0.98.62:35291/ using document root <none> and password file <none>
I20260929 19:00:51.916565 14079 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:51.916640 14079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:51.916795 14079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:51.917894 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "698ac314168e447e88b3114f904485a2"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-gfsd"
server_key: "39cf997cf759b5f353209e2904fc92c7"
server_key_iv: "095316e4dcd16860cfec4faada95e591"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.918258 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "698ac314168e447e88b3114f904485a2"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-gfsd"
server_key: "39cf997cf759b5f353209e2904fc92c7"
server_key_iv: "095316e4dcd16860cfec4faada95e591"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.919682 14079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:00:51.920497 14094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.920679 14079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:51.920782 14079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "698ac314168e447e88b3114f904485a2"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-gfsd"
server_key: "39cf997cf759b5f353209e2904fc92c7"
server_key_iv: "095316e4dcd16860cfec4faada95e591"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.920866 14079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:51.944911 14079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:51.947413 14079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:51.947579 14079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:51.951493 14079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260929 19:00:51.951547 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260929 19:00:51.951918 14079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:51.952893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14079
I20260929 19:00:51.953014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:51.953027 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:51.953244 392 external_mini_cluster.cc:1428] Setting key 13e5b356dd739fd9790ab4032ed6b8ed
I20260929 19:00:51.956192 14147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap starting.
I20260929 19:00:51.956911 14147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:51.957201 14147 log.cc:824] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:51.957953 14147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: No bootstrap required, opened a new log
I20260929 19:00:51.960111 14147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:51.960289 14147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:51.960325 14147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698ac314168e447e88b3114f904485a2, State: Initialized, Role: FOLLOWER
Sep 29 19:00:51 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:51.960449 14147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:51.960541 14147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:51.960572 14147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:51.960613 14147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:51.961292 14147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:51.961405 14147 leader_election.cc:302] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698ac314168e447e88b3114f904485a2; no voters:
I20260929 19:00:51.961591 14147 leader_election.cc:288] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:51.961623 14152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:51.961745 14152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 LEADER]: Becoming Leader. State: Replica: 698ac314168e447e88b3114f904485a2, State: Running, Role: LEADER
I20260929 19:00:51.961890 14147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:51.962280 14152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:51.962666 14154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698ac314168e447e88b3114f904485a2. Latest consensus state: current_term: 1 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:51.962738 14154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:51.962940 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:51.963248 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:51.963479 14160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:51.964049 14160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:51.964989 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.953813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.0.98.62:40939)
0929 19:00:51.953992 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.953999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.954023 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:00:51.954502 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.954508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.954556 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.954675 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.955481 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.956420 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.957520 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.958047 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.958176 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.960969 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.960973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.960973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.960983 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.963166 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.963622 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.963624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.963625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.963737 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.964026 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.964027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.964027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.964350 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.964509 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.964635 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.964789 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260929 19:00:51.965744 14160 catalog_manager.cc:1347] Generated new cluster ID: fd5e28e3cb204db29652b28ce7c95ab7
I20260929 19:00:51.965796 14160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:00:51.990545 14160 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:51.991120 14160 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:51.995966 14160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Generated new TSK 0
I20260929 19:00:51.996178 14160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:52.028575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.110211 14173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.110414 14173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.110437 14173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.111687 14173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.111739 14173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.111791 14173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:52.113349 14173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:52.113632 14173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.113865 14173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.118913 14181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.119037 14179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.118919 14178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.119134 14173 server_base.cc:1034] running on GCE node
I20260929 19:00:52.119426 14173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.119670 14173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.120849 14173 hybrid_clock.cc:648] HybridClock initialized: now 1790708452120838 us; error 41 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:52.123596 14173 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:52.124097 14173 webserver.cc:460] Webserver started at http://127.0.98.1:34903/ using document root <none> and password file <none>
I20260929 19:00:52.124408 14173 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:52.124480 14173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:52.124619 14173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:52.125676 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d17756bb3ea478b85ec6c316efa1008"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "40d17472ce550888e54f8b113a6ec9d8"
server_key_iv: "3646b3fc2fe6e45977dbafc63964672f"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.126060 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d17756bb3ea478b85ec6c316efa1008"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "40d17472ce550888e54f8b113a6ec9d8"
server_key_iv: "3646b3fc2fe6e45977dbafc63964672f"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.127542 14173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:52.128371 14188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.128541 14173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:52.128644 14173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "6d17756bb3ea478b85ec6c316efa1008"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "40d17472ce550888e54f8b113a6ec9d8"
server_key_iv: "3646b3fc2fe6e45977dbafc63964672f"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.128717 14173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:52.151368 14173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:52.153582 14173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:52.153756 14173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:52.154009 14173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:52.154318 14173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:52.154372 14173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.154412 14173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:52.154457 14173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.160567 14173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45441
I20260929 19:00:52.160621 14301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45441 every 8 connection(s)
I20260929 19:00:52.160979 14173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:52.163785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14173
I20260929 19:00:52.163919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:52.164131 392 external_mini_cluster.cc:1428] Setting key 6afb5e58e47f22a2cf65a13b1044e3f2
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:52.171850 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.161831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36943 (local address 127.0.98.62:40939)
0929 19:00:52.161926 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.161928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.162000 (+ 72us) server_negotiation.cc:407] Connection header received
0929 19:00:52.162554 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.162555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.162566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.162637 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.163738 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.164592 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.165742 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.166088 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.166168 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.168722 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.168723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.168724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.168732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.170306 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.170818 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.170819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.170819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.170878 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.171161 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.171161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.171161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.171259 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.171385 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.171696 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.171785 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:00:52.172145 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.161715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:36943)
0929 19:00:52.161866 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.161908 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.162452 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.162697 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.162700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.163419 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:52.163619 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.163624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.164689 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.164691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.165625 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.165628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.166206 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.166327 (+ 121us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.166493 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.167054 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:52.167064 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:52.168624 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:52.170382 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.170390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.170392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.170726 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.170949 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.170950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.170950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.171087 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.171384 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:52.171392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:52.171555 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.171676 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.171854 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":91,"threads_started":1}
I20260929 19:00:52.172710 14302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:52.172806 14302 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:52.172995 14302 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:52.173487 14111 ts_manager.cc:194] Registered new tserver with Master: 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:00:52.174165 14111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:52.190487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.271979 14307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.272212 14307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.272262 14307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.273409 14307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.273473 14307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.273517 14307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:52.274938 14307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:52.275199 14307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.275447 14307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.280601 14312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.280555 14313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.280646 14307 server_base.cc:1034] running on GCE node
W20260929 19:00:52.280599 14315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.281039 14307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.281303 14307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.282493 14307 hybrid_clock.cc:648] HybridClock initialized: now 1790708452282472 us; error 34 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:52.284969 14307 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:52.285370 14307 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260929 19:00:52.285586 14307 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:52.285661 14307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:52.285816 14307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:52.286816 14307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "9e9cf6c96601abf75f91439ceca22328"
server_key_iv: "c1627ef86befa175b722dbb7869c3e40"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.287175 14307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "9e9cf6c96601abf75f91439ceca22328"
server_key_iv: "c1627ef86befa175b722dbb7869c3e40"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.288503 14307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:52.289268 14322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.289461 14307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:52.289572 14307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "9e9cf6c96601abf75f91439ceca22328"
server_key_iv: "c1627ef86befa175b722dbb7869c3e40"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.289667 14307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:52.307539 14307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:52.309617 14307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:52.309810 14307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:52.310081 14307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:52.310420 14307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:52.310477 14307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.310518 14307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:52.310570 14307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.317282 14307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40329
I20260929 19:00:52.317359 14435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40329 every 8 connection(s)
I20260929 19:00:52.317720 14307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:52.325459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14307
I20260929 19:00:52.325619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:52.325836 392 external_mini_cluster.cc:1428] Setting key b4b6dce34c2b81dd75bb69b6c6880902
I20260929 19:00:52.327729 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.318528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39473 (local address 127.0.98.62:40939)
0929 19:00:52.318613 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.318616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.318790 (+ 174us) server_negotiation.cc:407] Connection header received
0929 19:00:52.319341 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.319342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.319351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.319409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.320400 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.320888 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.321867 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.322042 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.322092 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.324481 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.324483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.324484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.324490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.326147 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.326631 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.326632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.326632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.326693 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.327008 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.327009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.327009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.327107 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.327158 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.327595 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.327667 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 19:00:52.328042 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.318471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:39473)
0929 19:00:52.318653 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.318694 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.319197 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.319463 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.319466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.320077 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:52.320316 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.320322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.320972 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.320973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.321790 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.321793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.322141 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.322142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.322279 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.322750 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:52.322759 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:52.324385 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:52.326224 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.326231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.326234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.326542 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.326772 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.326773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.326774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.326935 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.327187 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:52.327194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:52.327414 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.327546 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.327744 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260929 19:00:52.328855 14436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:52.328951 14436 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:52.329139 14436 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:52.329543 14111 ts_manager.cc:194] Registered new tserver with Master: 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
I20260929 19:00:52.329928 14111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:52.352207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.434732 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.434968 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.435009 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.436312 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.436373 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.436420 14441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:52.437887 14441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:52.438154 14441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.438393 14441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.443527 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.443506 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.443600 14441 server_base.cc:1034] running on GCE node
W20260929 19:00:52.443509 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.444015 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.444205 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.445364 14441 hybrid_clock.cc:648] HybridClock initialized: now 1790708452445342 us; error 37 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:52.447968 14441 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:52.448326 14441 webserver.cc:460] Webserver started at http://127.0.98.3:40927/ using document root <none> and password file <none>
I20260929 19:00:52.448515 14441 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:52.448555 14441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:52.448638 14441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:52.449635 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "e509dbb02a6e4ef5916119d8e45f0259"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "c18ba922d414efe9789ed44a7211ff43"
server_key_iv: "0cc52f1d98f650dd140ed6c37f78efb7"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.449936 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e509dbb02a6e4ef5916119d8e45f0259"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "c18ba922d414efe9789ed44a7211ff43"
server_key_iv: "0cc52f1d98f650dd140ed6c37f78efb7"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.451139 14441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:00:52.452008 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.452196 14441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:52.452288 14441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "e509dbb02a6e4ef5916119d8e45f0259"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "c18ba922d414efe9789ed44a7211ff43"
server_key_iv: "0cc52f1d98f650dd140ed6c37f78efb7"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.452350 14441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:52.468560 14441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:52.470969 14441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:52.471081 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:52.471340 14441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:52.471666 14441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:52.471716 14441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.471747 14441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:52.471760 14441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.477756 14441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36499
I20260929 19:00:52.477828 14569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36499 every 8 connection(s)
I20260929 19:00:52.478119 14441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:52.487016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14441
I20260929 19:00:52.487133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:52.487375 392 external_mini_cluster.cc:1428] Setting key eba18308fe3ec5c352b4fe60583bd569
I20260929 19:00:52.487437 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.478905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42303 (local address 127.0.98.62:40939)
0929 19:00:52.479019 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.479021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.479206 (+ 185us) server_negotiation.cc:407] Connection header received
0929 19:00:52.479820 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.479821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.479833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.479878 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.480813 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.481355 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.482338 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.482551 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.482637 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.484703 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.484705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.484705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.484712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.485937 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.486295 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.486296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.486296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.486354 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.486631 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.486633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.486633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.486761 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.486886 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.487216 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.487329 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 19:00:52.487630 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.478817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:42303)
0929 19:00:52.479048 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.479094 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.479703 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.479958 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.479962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.480522 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:52.480712 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.480717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.481451 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.481452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.482247 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.482249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.482646 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.482647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.482779 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.483229 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:52.483237 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:52.484612 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:52.486011 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.486016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.486018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.486228 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.486413 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.486413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.486414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.486542 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.486883 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:52.486887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:52.486994 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.487076 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.487367 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":120,"threads_started":1}
I20260929 19:00:52.488232 14570 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:52.488360 14570 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:52.488556 14570 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:52.488909 14111 ts_manager.cc:194] Registered new tserver with Master: e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499)
I20260929 19:00:52.489290 14111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42303
I20260929 19:00:52.499626 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:52.507403 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.501025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.98.62:40939)
0929 19:00:52.501108 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.501110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.501206 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:00:52.501296 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.501297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.501308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.501380 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.501950 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.502481 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.503226 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.503476 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.503554 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.504775 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.504776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.504776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.504784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.506153 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.506549 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.506551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.506551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.506609 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.506831 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.506832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.506832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.506981 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.507091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.507210 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.507306 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 19:00:52.509302 14111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:52.511003 14111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:52.517578 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.0.1:41172)
0929 19:00:52.514017 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.514026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.514076 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.514338 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.514339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.514341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:52.514393 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.514395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515301 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.515303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.516475 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.516478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517277 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.517355 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517370 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.517448 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.517500 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":27,"thread_start_us":68,"threads_started":1}
I20260929 19:00:52.517590 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.98.2:40329)
0929 19:00:52.513970 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.513974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.514132 (+ 158us) server_negotiation.cc:407] Connection header received
0929 19:00:52.514174 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.514176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.514193 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.514261 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:52.514470 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515150 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.516567 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517194 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517268 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517323 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.517471 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.517510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
I20260929 19:00:52.517973 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36499 (local address 127.0.0.1:52742)
0929 19:00:52.513771 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.513802 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.513874 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.514153 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.514157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.514159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:52.514228 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.514230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515575 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.515576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.516801 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.516803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517748 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.517807 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517814 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.517866 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.517902 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260929 19:00:52.518035 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.98.3:36499)
0929 19:00:52.513879 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.513884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.513895 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:52.513965 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.513967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.513983 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.514061 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:52.514319 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515456 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.516900 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517585 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517711 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.517899 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.517950 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":87,"threads_started":1}
I20260929 19:00:52.518086 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45441 (local address 127.0.0.1:52020)
0929 19:00:52.513736 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.513791 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.513869 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.514178 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.514179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.514181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:52.514230 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.514231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515576 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.515576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.516801 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.516803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517815 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.517893 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517897 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.517948 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.517987 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":70,"thread_start_us":61,"threads_started":1}
I20260929 19:00:52.518694 14504 tablet_service.cc:1467] Processing CreateTablet for tablet c98ba413524c4024bc284403148f19f7 (DEFAULT_TABLE table=test-table [id=04edd6a2b882462e8d480b4f15a56787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:52.518703 14370 tablet_service.cc:1467] Processing CreateTablet for tablet c98ba413524c4024bc284403148f19f7 (DEFAULT_TABLE table=test-table [id=04edd6a2b882462e8d480b4f15a56787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:52.519047 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98ba413524c4024bc284403148f19f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:52.519055 14370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98ba413524c4024bc284403148f19f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:52.522498 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.513534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.0.98.1:45441)
0929 19:00:52.513791 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.513797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.513978 (+ 181us) server_negotiation.cc:407] Connection header received
0929 19:00:52.514019 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.514021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.514036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.514100 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:52.514318 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.515473 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.516890 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.517657 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.517726 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.517786 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.518208 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.518256 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
I20260929 19:00:52.522477 14587 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Bootstrap starting.
I20260929 19:00:52.522477 14588 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Bootstrap starting.
I20260929 19:00:52.523124 14236 tablet_service.cc:1467] Processing CreateTablet for tablet c98ba413524c4024bc284403148f19f7 (DEFAULT_TABLE table=test-table [id=04edd6a2b882462e8d480b4f15a56787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:52.523221 14587 tablet_bootstrap.cc:654] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:52.523391 14588 tablet_bootstrap.cc:654] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:52.523487 14236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98ba413524c4024bc284403148f19f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:52.523869 14587 log.cc:824] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:52.523973 14588 log.cc:824] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:52.524648 14587 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: No bootstrap required, opened a new log
I20260929 19:00:52.524744 14587 ts_tablet_manager.cc:1397] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:52.524781 14588 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: No bootstrap required, opened a new log
I20260929 19:00:52.524854 14588 ts_tablet_manager.cc:1397] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:52.526319 14587 raft_consensus.cc:348] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.526484 14587 raft_consensus.cc:374] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:52.526525 14587 raft_consensus.cc:729] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d792f9bf1bb4a30a813d832c8f58b50, State: Initialized, Role: FOLLOWER
I20260929 19:00:52.526633 14587 consensus_queue.cc:260] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.526810 14588 raft_consensus.cc:348] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.526927 14587 ts_tablet_manager.cc:1428] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:00:52.526976 14588 raft_consensus.cc:374] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:52.527020 14588 raft_consensus.cc:729] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e509dbb02a6e4ef5916119d8e45f0259, State: Initialized, Role: FOLLOWER
I20260929 19:00:52.527050 14436 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
I20260929 19:00:52.527122 14588 consensus_queue.cc:260] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.527418 14588 ts_tablet_manager.cc:1428] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:00:52.527431 14570 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
I20260929 19:00:52.527626 14591 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap starting.
I20260929 19:00:52.528322 14591 tablet_bootstrap.cc:654] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:52.528824 14591 log.cc:824] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:52.529531 14591 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: No bootstrap required, opened a new log
I20260929 19:00:52.529587 14591 ts_tablet_manager.cc:1397] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:52.531229 14591 raft_consensus.cc:348] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.531430 14591 raft_consensus.cc:374] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:52.531457 14591 raft_consensus.cc:729] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d17756bb3ea478b85ec6c316efa1008, State: Initialized, Role: FOLLOWER
I20260929 19:00:52.531553 14591 consensus_queue.cc:260] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.531785 14591 ts_tablet_manager.cc:1428] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:00:52.531834 14302 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
W20260929 19:00:52.568327 14437 tablet.cc:2369] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:52.661731 14303 tablet.cc:2369] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:52.728705 14571 tablet.cc:2369] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:52.822156 14592 raft_consensus.cc:482] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:52.822296 14592 raft_consensus.cc:504] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.822664 14592 leader_election.cc:288] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441)
I20260929 19:00:52.825560 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.822853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45441 (local address 127.0.98.2:50587)
0929 19:00:52.822913 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.822925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.822969 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.823136 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.823137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.823138 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:52.823228 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.823229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.823934 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.823935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.824737 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.824739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.825302 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.825303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.825381 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.825384 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.825445 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.825488 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 19:00:52.825560 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.822932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50587 (local address 127.0.98.1:45441)
0929 19:00:52.822971 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.822973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.822983 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:52.823040 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.823041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.823057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.823133 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:52.823315 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.823857 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.824846 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.825222 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.825304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.825314 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.825463 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.825496 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:00:52.825819 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.822931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48505 (local address 127.0.98.3:36499)
0929 19:00:52.823000 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.823002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.823119 (+ 117us) server_negotiation.cc:407] Connection header received
0929 19:00:52.823196 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.823197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.823214 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.823287 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:52.823475 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.823933 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.825062 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.825429 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.825513 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.825521 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.825704 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.825753 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:00:52.825912 14256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98ba413524c4024bc284403148f19f7" candidate_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d17756bb3ea478b85ec6c316efa1008" is_pre_election: true
I20260929 19:00:52.826032 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.822853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36499 (local address 127.0.98.2:48505)
0929 19:00:52.823033 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.823046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.823129 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.823327 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.823328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.823330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:52.823400 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.823402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.824030 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.824031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.824974 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.824977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.825514 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.825515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.825591 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.825594 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.825912 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.825955 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":13,"thread_start_us":72,"threads_started":1}
I20260929 19:00:52.826081 14256 raft_consensus.cc:2454] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d792f9bf1bb4a30a813d832c8f58b50 in term 0.
I20260929 19:00:52.826375 14324 leader_election.cc:302] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d792f9bf1bb4a30a813d832c8f58b50, 6d17756bb3ea478b85ec6c316efa1008; no voters:
I20260929 19:00:52.826390 14524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98ba413524c4024bc284403148f19f7" candidate_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509dbb02a6e4ef5916119d8e45f0259" is_pre_election: true
I20260929 19:00:52.826521 14524 raft_consensus.cc:2454] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d792f9bf1bb4a30a813d832c8f58b50 in term 0.
I20260929 19:00:52.826560 14592 raft_consensus.cc:2781] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:52.826644 14592 raft_consensus.cc:482] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:52.826676 14592 raft_consensus.cc:3037] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:52.827524 14592 raft_consensus.cc:504] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.827697 14592 leader_election.cc:288] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [CANDIDATE]: Term 1 election: Requested vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441)
I20260929 19:00:52.827818 14524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98ba413524c4024bc284403148f19f7" candidate_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509dbb02a6e4ef5916119d8e45f0259"
I20260929 19:00:52.827900 14256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98ba413524c4024bc284403148f19f7" candidate_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d17756bb3ea478b85ec6c316efa1008"
I20260929 19:00:52.827930 14524 raft_consensus.cc:3037] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:52.827996 14256 raft_consensus.cc:3037] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:52.828688 14256 raft_consensus.cc:2454] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d792f9bf1bb4a30a813d832c8f58b50 in term 1.
I20260929 19:00:52.828702 14524 raft_consensus.cc:2454] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d792f9bf1bb4a30a813d832c8f58b50 in term 1.
I20260929 19:00:52.828859 14324 leader_election.cc:302] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d792f9bf1bb4a30a813d832c8f58b50, 6d17756bb3ea478b85ec6c316efa1008; no voters:
I20260929 19:00:52.829016 14592 raft_consensus.cc:2781] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:52.829185 14592 raft_consensus.cc:686] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 LEADER]: Becoming Leader. State: Replica: 4d792f9bf1bb4a30a813d832c8f58b50, State: Running, Role: LEADER
I20260929 19:00:52.829278 14592 consensus_queue.cc:237] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:52.830057 14111 catalog_manager.cc:5495] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } health_report { overall_health: HEALTHY } } }
I20260929 19:00:52.871026 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.868552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.0.98.2:40329)
0929 19:00:52.868619 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.868621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.868636 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:00:52.868692 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.868693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.868713 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.868812 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:52.868946 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.869433 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.870217 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.870414 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.870489 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.870683 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:52.870784 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:52.870886 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.870907 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.870928 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:00:52.873040 14256 raft_consensus.cc:1261] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Refusing update from remote peer 4d792f9bf1bb4a30a813d832c8f58b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:52.873037 14524 raft_consensus.cc:1261] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Refusing update from remote peer 4d792f9bf1bb4a30a813d832c8f58b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:52.873430 14592 consensus_queue.cc:1035] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:52.873624 14597 consensus_queue.cc:1035] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:52.876698 14602 mvcc.cc:205] Tried to move back new op lower bound from 7334741822964817920 to 7334741822789242880. Current Snapshot: MvccSnapshot[applied={T|T < 7334741822964817920}]
I20260929 19:00:52.878907 14110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:52.879002 14110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:52.879822 14110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Sending DeleteTablet for 3 replicas of tablet c98ba413524c4024bc284403148f19f7
I20260929 19:00:52.880170 14504 tablet_service.cc:1514] Processing DeleteTablet for tablet c98ba413524c4024bc284403148f19f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52742
I20260929 19:00:52.880204 14370 tablet_service.cc:1514] Processing DeleteTablet for tablet c98ba413524c4024bc284403148f19f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41172
I20260929 19:00:52.880228 14236 tablet_service.cc:1514] Processing DeleteTablet for tablet c98ba413524c4024bc284403148f19f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52020
I20260929 19:00:52.880410 14611 tablet_replica.cc:330] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: stopping tablet replica
I20260929 19:00:52.880424 14610 tablet_replica.cc:330] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: stopping tablet replica
I20260929 19:00:52.880492 14611 raft_consensus.cc:2229] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:52.880487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14173
I20260929 19:00:52.880517 14610 raft_consensus.cc:2229] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:52.880607 14611 raft_consensus.cc:2258] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:52.880618 14610 raft_consensus.cc:2258] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:52.881072 14611 ts_tablet_manager.cc:1905] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:52.882654 14610 ts_tablet_manager.cc:1905] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:52.882843 14611 ts_tablet_manager.cc:1918] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:52.882916 14611 log.cc:1196] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/c98ba413524c4024bc284403148f19f7
I20260929 19:00:52.883163 14611 ts_tablet_manager.cc:1939] T c98ba413524c4024bc284403148f19f7 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting consensus metadata
I20260929 19:00:52.883587 14095 catalog_manager.cc:4841] TS 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): tablet c98ba413524c4024bc284403148f19f7 (table test-table [id=04edd6a2b882462e8d480b4f15a56787]) successfully deleted
I20260929 19:00:52.884615 14610 ts_tablet_manager.cc:1918] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:52.884692 14610 log.cc:1196] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/c98ba413524c4024bc284403148f19f7
I20260929 19:00:52.884955 14610 ts_tablet_manager.cc:1939] T c98ba413524c4024bc284403148f19f7 P e509dbb02a6e4ef5916119d8e45f0259: Deleting consensus metadata
I20260929 19:00:52.885541 14095 catalog_manager.cc:4841] TS e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): tablet c98ba413524c4024bc284403148f19f7 (table test-table [id=04edd6a2b882462e8d480b4f15a56787]) successfully deleted
W20260929 19:00:52.887274 14096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45441 (error 108)
I20260929 19:00:52.887475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14307
I20260929 19:00:52.888450 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.888326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45441 (local address 127.0.0.1:52032)
0929 19:00:52.888374 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.888410 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:00:52.888571 14096 catalog_manager.cc:4584] TS 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c98ba413524c4024bc284403148f19f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45441: connect: Connection refused (error 111)
I20260929 19:00:52.892562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14441
I20260929 19:00:52.897017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14079
I20260929 19:00:52.902873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=35291
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.984457 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.984660 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.984683 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.985873 14614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:52.985917 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.985931 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.985944 14614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:52.985957 14614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:52.987597 14614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:52.987852 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.988119 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.997884 14620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.997972 14621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.998121 14623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.998152 14614 server_base.cc:1034] running on GCE node
I20260929 19:00:52.998477 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.998792 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.999963 14614 hybrid_clock.cc:648] HybridClock initialized: now 1790708452999938 us; error 37 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.002702 14614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:53.003180 14614 webserver.cc:460] Webserver started at http://127.0.98.62:35291/ using document root <none> and password file <none>
I20260929 19:00:53.003432 14614 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.003490 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.004913 14614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.005678 14630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.005944 14614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.006031 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "698ac314168e447e88b3114f904485a2"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-gfsd"
server_key: "39cf997cf759b5f353209e2904fc92c7"
server_key_iv: "095316e4dcd16860cfec4faada95e591"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.006547 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.030110 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.032191 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.032378 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.036693 14614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260929 19:00:53.036770 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260929 19:00:53.037156 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:53.038058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14614
I20260929 19:00:53.038364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45441
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=34903
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:53.038935 14683 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:00:53.039736 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap starting.
I20260929 19:00:53.047935 14683 log.cc:824] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.049898 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:00:53.050200 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap complete.
I20260929 19:00:53.052556 14683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.052809 14683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698ac314168e447e88b3114f904485a2, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.052995 14683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.053071 14683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:53.053128 14683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:53.053197 14683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:00:53.054208 14683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.054327 14683 leader_election.cc:302] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698ac314168e447e88b3114f904485a2; no voters:
I20260929 19:00:53.054535 14683 leader_election.cc:288] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:00:53.054571 14687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:00:53.054699 14687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 LEADER]: Becoming Leader. State: Replica: 698ac314168e447e88b3114f904485a2, State: Running, Role: LEADER
I20260929 19:00:53.054802 14683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:53.054781 14687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.055105 14687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698ac314168e447e88b3114f904485a2. Latest consensus state: current_term: 2 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:53.055174 14687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:53.055334 14688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:53.055475 14688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:53.055414 14691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:53.057034 14691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:53.057134 14691 catalog_manager.cc:1259] Loaded cluster ID: fd5e28e3cb204db29652b28ce7c95ab7
I20260929 19:00:53.057161 14691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:00:53.057400 14691 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 19:00:53.060410 14691 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 19:00:53.064374 14691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:53.064599 14691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Loaded TSK: 0
I20260929 19:00:53.064834 14691 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:00:53.131928 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.132126 14685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.132149 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.133293 14685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.133334 14685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.133360 14685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:53.134794 14685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34903
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:53.135015 14685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.135248 14685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.141149 14711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.141140 14713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.141140 14710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.141438 14685 server_base.cc:1034] running on GCE node
I20260929 19:00:53.141641 14685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.141911 14685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.143078 14685 hybrid_clock.cc:648] HybridClock initialized: now 1790708453143064 us; error 31 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.145689 14685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:53.146142 14685 webserver.cc:460] Webserver started at http://127.0.98.1:34903/ using document root <none> and password file <none>
I20260929 19:00:53.146380 14685 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.146490 14685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.148048 14685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:00:53.148958 14720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.149116 14685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.149276 14685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "6d17756bb3ea478b85ec6c316efa1008"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "40d17472ce550888e54f8b113a6ec9d8"
server_key_iv: "3646b3fc2fe6e45977dbafc63964672f"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.149607 14685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.160686 14685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.162781 14685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.162930 14685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.163200 14685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:53.163739 14727 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:00:53.165742 14685 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:00:53.165835 14685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:00:53.165889 14685 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:00:53.166626 14685 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:00:53.166698 14685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:53.166747 14727 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap starting.
I20260929 19:00:53.174674 14685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45441
I20260929 19:00:53.175223 14685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:53.177892 14727 log.cc:824] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.178845 14834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45441 every 8 connection(s)
I20260929 19:00:53.180418 14727 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:00:53.180737 14727 tablet_bootstrap.cc:492] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap complete.
I20260929 19:00:53.181942 14727 ts_tablet_manager.cc:1397] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260929 19:00:53.183115 14727 raft_consensus.cc:348] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:53.183447 14727 raft_consensus.cc:729] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d17756bb3ea478b85ec6c316efa1008, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.183545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14685
I20260929 19:00:53.183612 14727 consensus_queue.cc:260] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260929 19:00:53.183899 14727 ts_tablet_manager.cc:1428] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260929 19:00:53.184239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40329
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=39181
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.190017 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.176013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:54331)
0929 19:00:53.176218 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.176262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.176747 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.178714 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.178718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.179168 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.179461 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.179468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.181148 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.181149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.182446 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.182449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.183255 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.183256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.183427 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.184268 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.184278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.185780 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.188152 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.188160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.188162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.188500 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.188808 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.188809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.188810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.188932 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.189335 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.189342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.189488 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.189601 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.189802 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260929 19:00:53.190030 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.176119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54331 (local address 127.0.98.62:40939)
0929 19:00:53.178198 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.178207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.178261 (+ 54us) server_negotiation.cc:407] Connection header received
0929 19:00:53.178401 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.178408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.178466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.178627 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.179593 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.181011 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.182547 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.183151 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.183303 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.185887 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.185891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.185891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.185901 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.188020 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.188609 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.188612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.188613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.188719 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.189010 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.189011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.189012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.189244 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.189340 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.189682 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.189820 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":92,"threads_started":1}
I20260929 19:00:53.190835 14835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:53.190958 14835 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:53.191157 14835 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:53.192003 14647 ts_manager.cc:194] Registered new tserver with Master: 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441)
I20260929 19:00:53.193159 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54331
I20260929 19:00:53.194284 14835 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
I20260929 19:00:53.200219 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.193475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.0.98.1:45441)
0929 19:00:53.193654 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.193658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.195521 (+ 1863us) server_negotiation.cc:407] Connection header received
0929 19:00:53.195625 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.195627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.195663 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.195774 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.195982 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.196734 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.197931 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.199239 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.199474 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.199677 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.199929 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.200006 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260929 19:00:53.200546 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.192611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45441 (local address 127.0.0.1:52044)
0929 19:00:53.195413 (+ 2802us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.195445 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.195542 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.195781 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.195784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.195786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.195900 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.195902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.196833 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.196834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.197744 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.197746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.199398 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.199399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.199744 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.199768 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.199871 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.200409 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2765,"thread_start_us":149,"threads_started":1}
I20260929 19:00:53.201933 14750 tablet_service.cc:1514] Processing DeleteTablet for tablet c98ba413524c4024bc284403148f19f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52044
I20260929 19:00:53.202499 14844 tablet_replica.cc:330] stopping tablet replica
I20260929 19:00:53.202636 14844 raft_consensus.cc:2229] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:53.202716 14844 raft_consensus.cc:2258] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.203038 14844 ts_tablet_manager.cc:1905] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.205358 14844 ts_tablet_manager.cc:1918] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.205456 14844 log.cc:1196] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/c98ba413524c4024bc284403148f19f7
I20260929 19:00:53.205703 14844 ts_tablet_manager.cc:1939] T c98ba413524c4024bc284403148f19f7 P 6d17756bb3ea478b85ec6c316efa1008: Deleting consensus metadata
I20260929 19:00:53.206315 14632 catalog_manager.cc:4841] TS 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441): tablet c98ba413524c4024bc284403148f19f7 (table test-table [id=04edd6a2b882462e8d480b4f15a56787]) successfully deleted
W20260929 19:00:53.276651 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.276880 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.276902 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.277973 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.278030 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.278059 14841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:53.279464 14841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39181
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:53.279680 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.279896 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.285013 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.285068 14841 server_base.cc:1034] running on GCE node
W20260929 19:00:53.284989 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.284977 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.285408 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.285668 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.286846 14841 hybrid_clock.cc:648] HybridClock initialized: now 1790708453286825 us; error 38 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.289288 14841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:53.289783 14841 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260929 19:00:53.290020 14841 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.290073 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.291469 14841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.292079 14859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.292335 14841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.292415 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "9e9cf6c96601abf75f91439ceca22328"
server_key_iv: "c1627ef86befa175b722dbb7869c3e40"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.292799 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.302459 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.305011 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.305161 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.305428 14841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:53.305748 14841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:53.305807 14841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.305850 14841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:53.305878 14841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.312600 14841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40329
I20260929 19:00:53.312672 14972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40329 every 8 connection(s)
I20260929 19:00:53.313056 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.320446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14841
I20260929 19:00:53.320839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36499
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=40927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:53.323498 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.313849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51303 (local address 127.0.98.62:40939)
0929 19:00:53.313910 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.313912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.314145 (+ 233us) server_negotiation.cc:407] Connection header received
0929 19:00:53.314716 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.314717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.314734 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.314790 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.315823 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.316469 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.317492 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.317711 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.317799 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.320195 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.320196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.320197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.320256 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.321971 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.322475 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.322476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.322476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.322530 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.322819 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.322819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.322819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.322921 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.322972 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.323361 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.323432 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:00:53.323777 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.313856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:51303)
0929 19:00:53.314023 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.314058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.314572 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.314864 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.314867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.315502 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.315720 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.315726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.316574 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.316577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.317388 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.317389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.317807 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.317808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.317928 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.318416 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.318425 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.320037 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.322058 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.322064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.322066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.322392 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.322598 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.322599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.322600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.322745 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.322985 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.322991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.323157 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.323321 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.323501 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260929 19:00:53.324281 14973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:53.324407 14973 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:53.324604 14973 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:53.325037 14647 ts_manager.cc:194] Registered new tserver with Master: 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
I20260929 19:00:53.325593 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51303
W20260929 19:00:53.402619 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.402814 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.402837 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.404066 14976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.404107 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.404134 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:53.405555 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40927
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:53.405786 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.406018 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.411201 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.411197 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.411198 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.411577 14976 server_base.cc:1034] running on GCE node
I20260929 19:00:53.411748 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.411939 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.413089 14976 hybrid_clock.cc:648] HybridClock initialized: now 1790708453413076 us; error 27 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.415585 14976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:53.415974 14976 webserver.cc:460] Webserver started at http://127.0.98.3:40927/ using document root <none> and password file <none>
I20260929 19:00:53.416193 14976 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.416303 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.417603 14976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:00:53.418236 14991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.418421 14976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:53.418527 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "e509dbb02a6e4ef5916119d8e45f0259"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "c18ba922d414efe9789ed44a7211ff43"
server_key_iv: "0cc52f1d98f650dd140ed6c37f78efb7"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.418860 14976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.425973 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.428315 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.428470 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.428689 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:53.428983 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:53.429044 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.429085 14976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:53.429112 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.434859 14976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36499
I20260929 19:00:53.434927 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36499 every 8 connection(s)
I20260929 19:00:53.435339 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:53.436578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 14976
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.444406 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.437000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.0.98.62:40939)
0929 19:00:53.437101 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.437102 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.437146 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:00:53.437208 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.437208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.437217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.437275 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.438035 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.438740 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.439798 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.439983 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.440036 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.441551 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.441552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.441552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.441558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.443114 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.443499 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.443500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.443501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.443567 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.443817 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.443817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.443818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.443944 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.444112 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.444182 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.444334 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260929 19:00:53.446043 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.436075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47587 (local address 127.0.98.62:40939)
0929 19:00:53.436130 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.436133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.436378 (+ 245us) server_negotiation.cc:407] Connection header received
0929 19:00:53.437061 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.437063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.437075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.437133 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.438297 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.438914 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.440321 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.440601 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.440652 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.443091 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.443093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.443093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.443101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.444877 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.445318 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.445319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.445319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.445368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.445571 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.445572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.445572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.445654 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.445697 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.445931 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.445988 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:00:53.446195 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.436036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:47587)
0929 19:00:53.436175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.436279 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.436946 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.437220 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.437224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.437986 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.438216 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.438221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.438994 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.438994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.440241 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.440244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.440681 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.440682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.440841 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.441329 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.441339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.442999 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.444943 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.444949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.444951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.445246 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.445429 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.445430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.445430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.445523 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.445702 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.445706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.445822 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.445902 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.446012 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260929 19:00:53.446635 15105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40939
I20260929 19:00:53.446753 15105 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:53.446930 15105 heartbeater.cc:511] Master 127.0.98.62:40939 requested a full tablet report, sending...
I20260929 19:00:53.447413 14647 ts_manager.cc:194] Registered new tserver with Master: e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499)
I20260929 19:00:53.447785 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47587
I20260929 19:00:53.456207 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:53.458892 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.456563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.98.62:40939)
0929 19:00:53.456604 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.456606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.456732 (+ 126us) server_negotiation.cc:407] Connection header received
0929 19:00:53.456802 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.456803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.456834 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.456906 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:53.457012 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.457476 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.458256 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.458445 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.458506 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.458627 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:53.458725 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:53.458788 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.458824 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.458839 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:00:53.459205 14647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:53.460975 14647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:53.463264 14750 tablet_service.cc:1467] Processing CreateTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 (DEFAULT_TABLE table=test-table [id=bc05a1c9be194a8bb7103eee37ca63ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.463533 14750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a8fcd0585f47258f3c37acd9b57fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.465018 14727 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap starting.
I20260929 19:00:53.465389 14727 tablet_bootstrap.cc:654] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.466005 14727 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: No bootstrap required, opened a new log
I20260929 19:00:53.466071 14727 ts_tablet_manager.cc:1397] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.466207 14727 raft_consensus.cc:348] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.466318 14727 raft_consensus.cc:374] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.466375 14727 raft_consensus.cc:729] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d17756bb3ea478b85ec6c316efa1008, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.466428 14727 consensus_queue.cc:260] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.466535 14727 ts_tablet_manager.cc:1428] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:53.467422 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.462945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36499 (local address 127.0.0.1:52744)
0929 19:00:53.462983 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.462994 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.463030 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.463552 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.463553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.463554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.463594 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.463596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.465073 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.465074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.466208 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.466211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.467150 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.467151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.467213 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.467217 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.467301 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.467349 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 19:00:53.467449 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.463057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.98.3:36499)
0929 19:00:53.463367 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.463372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.463384 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:00:53.463426 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.463428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.463445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.463538 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.463684 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.464936 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.466325 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.467057 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.467117 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.467177 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.467323 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.467364 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260929 19:00:53.467684 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.463153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.0.98.2:40329)
0929 19:00:53.463379 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.463383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.463392 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:53.463427 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.463429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.463445 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.463538 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.463831 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.465067 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.466303 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.467058 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.467138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.467197 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.467467 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.467519 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260929 19:00:53.467710 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.463058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.0.1:41188)
0929 19:00:53.463252 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.463263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.463338 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.463704 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.463705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.463707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.463745 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.463746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.465164 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.465165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.466214 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.466215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.467232 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.467233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.467317 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.467322 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.467373 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.467405 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":10,"thread_start_us":127,"threads_started":1}
I20260929 19:00:53.468165 15039 tablet_service.cc:1467] Processing CreateTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 (DEFAULT_TABLE table=test-table [id=bc05a1c9be194a8bb7103eee37ca63ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.468506 15039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a8fcd0585f47258f3c37acd9b57fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.468447 14907 tablet_service.cc:1467] Processing CreateTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 (DEFAULT_TABLE table=test-table [id=bc05a1c9be194a8bb7103eee37ca63ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.468698 14907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a8fcd0585f47258f3c37acd9b57fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.471390 15115 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Bootstrap starting.
I20260929 19:00:53.471418 15116 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Bootstrap starting.
I20260929 19:00:53.472003 15116 tablet_bootstrap.cc:654] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.472009 15115 tablet_bootstrap.cc:654] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.472509 15115 log.cc:824] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.472514 15116 log.cc:824] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.473331 15115 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: No bootstrap required, opened a new log
I20260929 19:00:53.473389 15116 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: No bootstrap required, opened a new log
I20260929 19:00:53.473435 15116 ts_tablet_manager.cc:1397] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:53.473424 15115 ts_tablet_manager.cc:1397] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:53.474757 15116 raft_consensus.cc:348] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.474812 15115 raft_consensus.cc:348] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.474918 15116 raft_consensus.cc:374] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.474933 15115 raft_consensus.cc:374] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.474964 15116 raft_consensus.cc:729] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d792f9bf1bb4a30a813d832c8f58b50, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.475009 15115 raft_consensus.cc:729] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e509dbb02a6e4ef5916119d8e45f0259, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.475100 15116 consensus_queue.cc:260] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.475135 15115 consensus_queue.cc:260] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.475430 15116 ts_tablet_manager.cc:1428] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:00:53.475445 14973 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
I20260929 19:00:53.475468 15115 ts_tablet_manager.cc:1428] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:00:53.475479 15105 heartbeater.cc:503] Master 127.0.98.62:40939 was elected leader, sending a full tablet report...
W20260929 19:00:53.563680 14974 tablet.cc:2369] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:53.679416 14836 tablet.cc:2369] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:53.680470 14840 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:53.680583 14840 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.680944 14840 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
I20260929 19:00:53.684216 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.681179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36499 (local address 127.0.98.1:33625)
0929 19:00:53.681236 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.681250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.681321 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.681584 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.681585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.681586 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.681653 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.681655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.682360 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.682361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.683165 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.683166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.683952 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.683953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.684034 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.684036 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.684100 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.684144 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:00:53.684218 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.681362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33625 (local address 127.0.98.3:36499)
0929 19:00:53.681424 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.681427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.681437 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:53.681467 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.681468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.681480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.681532 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.681779 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.682257 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.683475 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.683864 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.683954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.683965 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.684121 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.684162 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:00:53.684314 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.681395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43361 (local address 127.0.98.2:40329)
0929 19:00:53.681444 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.681446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.681460 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:00:53.681546 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.681547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.681559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.681616 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.682040 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.682508 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.683570 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.683941 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.684024 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.684035 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.684203 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.684241 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:00:53.684331 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.681293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.98.1:43361)
0929 19:00:53.681414 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.681424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.681477 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.681634 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.681634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.681636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.681955 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.681956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.682600 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.682601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.683476 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.683478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.684044 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.684046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.684115 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.684118 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.684194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.684247 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":10,"thread_start_us":66,"threads_started":1}
I20260929 19:00:53.684571 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a8fcd0585f47258f3c37acd9b57fd0" candidate_uuid: "6d17756bb3ea478b85ec6c316efa1008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509dbb02a6e4ef5916119d8e45f0259" is_pre_election: true
I20260929 19:00:53.684595 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a8fcd0585f47258f3c37acd9b57fd0" candidate_uuid: "6d17756bb3ea478b85ec6c316efa1008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" is_pre_election: true
I20260929 19:00:53.684725 14927 raft_consensus.cc:2454] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d17756bb3ea478b85ec6c316efa1008 in term 0.
I20260929 19:00:53.684725 15059 raft_consensus.cc:2454] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d17756bb3ea478b85ec6c316efa1008 in term 0.
I20260929 19:00:53.684989 14721 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008, e509dbb02a6e4ef5916119d8e45f0259; no voters:
I20260929 19:00:53.685128 14840 raft_consensus.cc:2781] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:53.685186 14840 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:53.685217 14840 raft_consensus.cc:3037] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.685885 14840 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
W20260929 19:00:53.685925 15106 tablet.cc:2369] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:53.686019 14840 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 1 election: Requested vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
I20260929 19:00:53.686208 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a8fcd0585f47258f3c37acd9b57fd0" candidate_uuid: "6d17756bb3ea478b85ec6c316efa1008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
I20260929 19:00:53.686226 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a8fcd0585f47258f3c37acd9b57fd0" candidate_uuid: "6d17756bb3ea478b85ec6c316efa1008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509dbb02a6e4ef5916119d8e45f0259"
I20260929 19:00:53.686290 14927 raft_consensus.cc:3037] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.686312 15059 raft_consensus.cc:3037] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.686977 14927 raft_consensus.cc:2454] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d17756bb3ea478b85ec6c316efa1008 in term 1.
I20260929 19:00:53.687057 15059 raft_consensus.cc:2454] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d17756bb3ea478b85ec6c316efa1008 in term 1.
I20260929 19:00:53.687141 14721 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d792f9bf1bb4a30a813d832c8f58b50, 6d17756bb3ea478b85ec6c316efa1008; no voters:
I20260929 19:00:53.687250 14840 raft_consensus.cc:2781] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:53.687434 14840 raft_consensus.cc:686] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 LEADER]: Becoming Leader. State: Replica: 6d17756bb3ea478b85ec6c316efa1008, State: Running, Role: LEADER
I20260929 19:00:53.687511 14840 consensus_queue.cc:237] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:53.688333 14647 catalog_manager.cc:5495] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6d17756bb3ea478b85ec6c316efa1008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } health_report { overall_health: HEALTHY } } }
I20260929 19:00:53.695441 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.693337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.98.1:45441)
0929 19:00:53.693374 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.693376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.693387 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:53.693423 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.693424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.693440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.693500 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:53.693598 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.694042 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.694736 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.694909 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.694957 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.695105 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:53.695236 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:53.695342 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.695362 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.695390 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:00:53.696925 15059 raft_consensus.cc:1261] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Refusing update from remote peer 6d17756bb3ea478b85ec6c316efa1008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:53.696954 14927 raft_consensus.cc:1261] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Refusing update from remote peer 6d17756bb3ea478b85ec6c316efa1008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:53.697268 15122 consensus_queue.cc:1035] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:53.697409 14840 consensus_queue.cc:1035] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:53.701560 14646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:53.701654 14646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:53.702391 14646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Sending DeleteTablet for 3 replicas of tablet 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:53.702713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14685
I20260929 19:00:53.702759 14750 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52044
I20260929 19:00:53.702816 14907 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41188
I20260929 19:00:53.702883 15039 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a8fcd0585f47258f3c37acd9b57fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52744
I20260929 19:00:53.703078 15135 tablet_replica.cc:330] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: stopping tablet replica
I20260929 19:00:53.703168 15135 raft_consensus.cc:2229] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:53.703383 15135 raft_consensus.cc:2258] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.703559 15134 tablet_replica.cc:330] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: stopping tablet replica
I20260929 19:00:53.703650 15134 raft_consensus.cc:2229] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:53.703744 15134 raft_consensus.cc:2258] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.704098 15135 ts_tablet_manager.cc:1905] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.704129 15134 ts_tablet_manager.cc:1905] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.706039 15134 ts_tablet_manager.cc:1918] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.706096 15134 log.cc:1196] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:53.706087 15135 ts_tablet_manager.cc:1918] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.706148 15135 log.cc:1196] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:53.706334 15134 ts_tablet_manager.cc:1939] T 23a8fcd0585f47258f3c37acd9b57fd0 P 4d792f9bf1bb4a30a813d832c8f58b50: Deleting consensus metadata
I20260929 19:00:53.706424 15135 ts_tablet_manager.cc:1939] T 23a8fcd0585f47258f3c37acd9b57fd0 P e509dbb02a6e4ef5916119d8e45f0259: Deleting consensus metadata
I20260929 19:00:53.706664 14631 catalog_manager.cc:4841] TS 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): tablet 23a8fcd0585f47258f3c37acd9b57fd0 (table test-table [id=bc05a1c9be194a8bb7103eee37ca63ed]) successfully deleted
I20260929 19:00:53.706802 14631 catalog_manager.cc:4841] TS e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): tablet 23a8fcd0585f47258f3c37acd9b57fd0 (table test-table [id=bc05a1c9be194a8bb7103eee37ca63ed]) successfully deleted
W20260929 19:00:53.708262 14632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45441 (error 108)
I20260929 19:00:53.708765 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.708599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45441 (local address 127.0.0.1:52064)
0929 19:00:53.708685 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.708726 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:00:53.708770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14841
W20260929 19:00:53.709445 14632 catalog_manager.cc:4584] TS 6d17756bb3ea478b85ec6c316efa1008 (127.0.98.1:45441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23a8fcd0585f47258f3c37acd9b57fd0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45441: connect: Connection refused (error 111)
I20260929 19:00:53.713517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14976
I20260929 19:00:53.717979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 14614
I20260929 19:00:53.722977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=35291
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:53.804687 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.804984 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.805027 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.807037 15137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:53.807116 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.807152 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.807178 15137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:53.807209 15137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:53.809916 15137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:53.810209 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.810504 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.815598 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.815788 15137 server_base.cc:1034] running on GCE node
W20260929 19:00:53.815562 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.815562 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.816280 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.816524 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.817662 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790708453817647 us; error 29 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.820024 15137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:53.820398 15137 webserver.cc:460] Webserver started at http://127.0.98.62:35291/ using document root <none> and password file <none>
I20260929 19:00:53.820569 15137 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.820608 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.822295 15137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:00:53.823132 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.823392 15137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:53.823494 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "698ac314168e447e88b3114f904485a2"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-gfsd"
server_key: "39cf997cf759b5f353209e2904fc92c7"
server_key_iv: "095316e4dcd16860cfec4faada95e591"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.823985 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.850435 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.852463 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.852581 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.856881 15137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260929 19:00:53.857015 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260929 19:00:53.857312 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:53.858022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15137
I20260929 19:00:53.858363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45441
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=34903
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:53.858979 15205 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:00:53.859973 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap starting.
I20260929 19:00:53.868655 15205 log.cc:824] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.871254 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:00:53.871665 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2: Bootstrap complete.
I20260929 19:00:53.873953 15205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.874169 15205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698ac314168e447e88b3114f904485a2, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.874356 15205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.874441 15205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:53.874495 15205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:53.874550 15205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:00:53.875622 15205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.875756 15205 leader_election.cc:302] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698ac314168e447e88b3114f904485a2; no voters:
I20260929 19:00:53.875993 15205 leader_election.cc:288] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:00:53.876029 15209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:00:53.876139 15209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [term 3 LEADER]: Becoming Leader. State: Replica: 698ac314168e447e88b3114f904485a2, State: Running, Role: LEADER
I20260929 19:00:53.876226 15209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260929 19:00:53.876286 15205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:53.876488 15209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698ac314168e447e88b3114f904485a2. Latest consensus state: current_term: 3 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:53.876560 15209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:53.876492 15210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "698ac314168e447e88b3114f904485a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698ac314168e447e88b3114f904485a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260929 19:00:53.876731 15210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698ac314168e447e88b3114f904485a2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:53.876765 15213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:53.877604 15213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:53.877734 15213 catalog_manager.cc:1259] Loaded cluster ID: fd5e28e3cb204db29652b28ce7c95ab7
I20260929 19:00:53.877780 15213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 19:00:53.878032 15213 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe1fe2ae9ad google::LogMessage::Fail() at ??:0
@ 0x7fe1fe2b188a google::LogMessage::SendToLog() at ??:0
@ 0x7fe1fe2ae4de google::LogMessage::Flush() at ??:0
@ 0x7fe1fe2afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe2061cf0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe2061d88b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe20017c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe200171954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe200fd56db start_thread at ??:0
@ 0x7fe1fcdac61f clone at ??:0
W20260929 19:00:53.949385 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.949604 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.949641 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.950757 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.950825 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.950872 15207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:53.952389 15207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34903
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:53.952654 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.952901 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.958041 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.958081 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.958273 15244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.958186 15207 server_base.cc:1034] running on GCE node
I20260929 19:00:53.958621 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.958881 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.960055 15207 hybrid_clock.cc:648] HybridClock initialized: now 1790708453960031 us; error 39 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.962656 15207 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:53.963157 15207 webserver.cc:460] Webserver started at http://127.0.98.1:34903/ using document root <none> and password file <none>
I20260929 19:00:53.963418 15207 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.963490 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.964879 15207 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:53.965581 15251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.965747 15207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:53.965844 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "6d17756bb3ea478b85ec6c316efa1008"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "40d17472ce550888e54f8b113a6ec9d8"
server_key_iv: "3646b3fc2fe6e45977dbafc63964672f"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.966197 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.986016 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.988533 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.988710 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.988956 15207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:53.989423 15258 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:00:53.991379 15207 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:00:53.991444 15207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:00:53.991488 15207 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:00:53.992115 15207 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:00:53.992169 15207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:53.992254 15258 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap starting.
I20260929 19:00:53.999435 15207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45441
I20260929 19:00:53.999478 15365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45441 every 8 connection(s)
I20260929 19:00:53.999904 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:54.003268 15258 log.cc:824] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:54.003913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15207
I20260929 19:00:54.004207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40329
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=39181
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:00:54.005659 15258 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:00:54.005975 15258 tablet_bootstrap.cc:492] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Bootstrap complete.
I20260929 19:00:54.007082 15258 ts_tablet_manager.cc:1397] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260929 19:00:54.008119 15258 raft_consensus.cc:348] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:54.008363 15258 raft_consensus.cc:729] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d17756bb3ea478b85ec6c316efa1008, State: Initialized, Role: FOLLOWER
I20260929 19:00:54.008502 15258 consensus_queue.cc:260] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:54.008785 15258 ts_tablet_manager.cc:1428] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260929 19:00:54.099514 15368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:00:54.000569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:40185)
0929 19:00:54.000740 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.000783 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.001366 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.099260 (+ 97894us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
W20260929 19:00:54.099817 15366 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40939: Client connection negotiation failed: client connection to 127.0.98.62:40939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:00:54.100207 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.100103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:60231)
0929 19:00:54.100140 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.100178 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 19:00:54.100587 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.100492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:49349)
0929 19:00:54.100536 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.100562 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:00:54.100721 15366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:00:54.109830 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:54.110054 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:54.110080 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:54.111375 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:54.111424 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:54.111455 15370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:54.113026 15370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39181
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:54.113307 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:54.113564 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:54.118796 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:54.118867 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:54.118796 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:54.119483 15370 server_base.cc:1034] running on GCE node
I20260929 19:00:54.119755 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:54.120004 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:54.121233 15370 hybrid_clock.cc:648] HybridClock initialized: now 1790708454121156 us; error 47 us; skew 500 ppm
Sep 29 19:00:54 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:54.124013 15370 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:54.124531 15370 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260929 19:00:54.124799 15370 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:54.124873 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:54.126322 15370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:54.126946 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.127326 15370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:54.127452 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "4d792f9bf1bb4a30a813d832c8f58b50"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "9e9cf6c96601abf75f91439ceca22328"
server_key_iv: "c1627ef86befa175b722dbb7869c3e40"
server_key_version: "encryptionkey@0"
I20260929 19:00:54.127816 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:54.145421 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:54.147503 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:54.147665 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:54.147899 15370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:54.148216 15370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:54.148272 15370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.148315 15370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:54.148342 15370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.154256 15370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40329
I20260929 19:00:54.154345 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40329 every 8 connection(s)
I20260929 19:00:54.154769 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:54.156005 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.155501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:40287)
0929 19:00:54.155724 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.155837 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260929 19:00:54.156456 15500 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40939: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
I20260929 19:00:54.156751 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.156656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:42271)
0929 19:00:54.156704 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.156730 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:00:54.157058 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.157001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:40579)
0929 19:00:54.157026 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.157045 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260929 19:00:54.157176 15500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:00:54.161064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15370
I20260929 19:00:54.161304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36499
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40939
--webserver_port=40927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:54.246021 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:54.246238 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:54.246269 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:54.247421 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:54.247464 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:54.247493 15503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:54.248878 15503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40927
--tserver_master_addrs=127.0.98.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:00:54.249063 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:54.249253 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:54.254412 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:54.254586 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:54.254630 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:54.254771 15503 server_base.cc:1034] running on GCE node
I20260929 19:00:54.254976 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:54.255165 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:54.256372 15503 hybrid_clock.cc:648] HybridClock initialized: now 1790708454256363 us; error 45 us; skew 500 ppm
Sep 29 19:00:54 dist-test-slave-gfsd krb5kdc[14066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:54.258922 15503 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:54.259384 15503 webserver.cc:460] Webserver started at http://127.0.98.3:40927/ using document root <none> and password file <none>
I20260929 19:00:54.259577 15503 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:54.259620 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:54.260860 15503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:54.261497 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.261680 15503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:54.261791 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "e509dbb02a6e4ef5916119d8e45f0259"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-gfsd"
server_key: "c18ba922d414efe9789ed44a7211ff43"
server_key_iv: "0cc52f1d98f650dd140ed6c37f78efb7"
server_key_version: "encryptionkey@0"
I20260929 19:00:54.262163 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:54.283489 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:54.285806 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:54.285965 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:54.286208 15503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:54.286509 15503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:54.286562 15503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.286603 15503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:54.286631 15503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.292589 15503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36499
I20260929 19:00:54.292632 15631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36499 every 8 connection(s)
I20260929 19:00:54.292958 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:54.293970 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.293604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:34587)
0929 19:00:54.293757 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.293830 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":115,"threads_started":1}
W20260929 19:00:54.294404 15632 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40939: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
I20260929 19:00:54.294714 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.294603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:44309)
0929 19:00:54.294646 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.294672 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:00:54.294981 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.294908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:58405)
0929 19:00:54.294947 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.294967 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:00:54.295094 15632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:00:54.296382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15503
I20260929 19:00:55.101351 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.101056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:38221)
0929 19:00:55.101262 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.101298 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":66,"threads_started":1}
I20260929 19:00:55.157779 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.157540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:36623)
0929 19:00:55.157664 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.157716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260929 19:00:55.295799 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.295553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:53245)
0929 19:00:55.295702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.295745 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260929 19:00:55.537737 15639 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:55.537878 15639 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:55.538143 15639 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
Sep 29 19:00:55 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 19:00:55 dist-test-slave-gfsd krb5kdc[14066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 19:00:55.549685 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.538602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42877 (local address 127.0.98.2:40329)
0929 19:00:55.538805 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:55.538813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:55.538840 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:00:55.539402 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:55.539406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:55.539477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:55.539626 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:55.540378 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.541432 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542410 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.542815 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542944 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.545344 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:55.545348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:55.545349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:55.545366 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:55.547491 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548126 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.548129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.548129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.548272 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548645 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.548646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.548646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.548961 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:55.549047 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:55.549435 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 19:00:55.549566 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260929 19:00:55.549711 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.538477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41543 (local address 127.0.98.3:36499)
0929 19:00:55.538654 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:55.538665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:55.538719 (+ 54us) server_negotiation.cc:407] Connection header received
0929 19:00:55.539402 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:55.539406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:55.539471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:55.539626 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:55.540377 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.541370 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542411 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.542812 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542932 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.545721 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:55.545725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:55.545725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:55.545735 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:55.547729 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548189 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.548192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.548192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.548282 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548645 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.548646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.548646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.548961 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:55.549047 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:55.549435 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 19:00:55.549566 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260929 19:00:55.549706 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.538513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.98.1:42877)
0929 19:00:55.538646 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.538658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:55.538727 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:55.539770 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:55.539771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:55.540027 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:55.540254 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.540261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.541539 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.541540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.542319 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.542321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.542919 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.543029 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.543729 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:55.543734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:55.545215 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:55.547598 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.547605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.547606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.548033 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.548354 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.548355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.548565 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.549045 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:55.549053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:55.549292 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 19:00:55.549374 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:00:55.549574 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":64,"thread_start_us":66,"threads_started":1}
I20260929 19:00:55.550248 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.538362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36499 (local address 127.0.98.1:41543)
0929 19:00:55.538422 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.538435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:55.538494 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:55.539667 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:55.539669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:55.540040 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:55.540255 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.540261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.541488 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.541489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.542319 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.542321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.542898 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.542899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.543020 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.543718 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:55.543726 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:55.545601 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:55.547806 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.547807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.547808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.548102 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.548354 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.548355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.548355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.548565 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.549045 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:55.549053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:55.549222 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 19:00:55.549356 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 19:00:55.549633 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":151}
W20260929 19:00:55.551097 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
W20260929 19:00:55.551178 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:55.551198 15252 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008; no voters: 4d792f9bf1bb4a30a813d832c8f58b50, e509dbb02a6e4ef5916119d8e45f0259
I20260929 19:00:55.551338 15639 raft_consensus.cc:2726] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:00:56.102082 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.101848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:49097)
0929 19:00:56.101998 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.102037 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260929 19:00:56.158615 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.158392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:34245)
0929 19:00:56.158534 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.158574 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260929 19:00:56.297076 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.296851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:59495)
0929 19:00:56.296993 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.297034 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260929 19:00:57.102823 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:57.102575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:59687)
0929 19:00:57.102737 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:57.102776 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260929 19:00:57.159446 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:57.159133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:42525)
0929 19:00:57.159352 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:57.159395 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260929 19:00:57.297865 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:57.297606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:34915)
0929 19:00:57.297773 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:57.297817 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260929 19:00:57.389422 15649 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:57.389537 15649 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:57.389719 15649 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
W20260929 19:00:57.390190 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
W20260929 19:00:57.390269 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:57.390286 15252 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008; no voters: 4d792f9bf1bb4a30a813d832c8f58b50, e509dbb02a6e4ef5916119d8e45f0259
I20260929 19:00:57.390385 15649 raft_consensus.cc:2726] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:00:58.103638 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.103429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:44295)
0929 19:00:58.103557 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.103593 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260929 19:00:58.160151 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.159918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:46423)
0929 19:00:58.160068 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.160109 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260929 19:00:58.298628 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.298408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:37339)
0929 19:00:58.298535 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.298576 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260929 19:00:58.976497 15653 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:58.976631 15653 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:00:58.976856 15653 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
W20260929 19:00:58.977401 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
W20260929 19:00:58.977473 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:00:58.977490 15252 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008; no voters: 4d792f9bf1bb4a30a813d832c8f58b50, e509dbb02a6e4ef5916119d8e45f0259
I20260929 19:00:58.977560 15653 raft_consensus.cc:2726] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:00:59.104420 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:59.104179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:41867)
0929 19:00:59.104333 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:59.104374 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260929 19:00:59.160964 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:59.160698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:33495)
0929 19:00:59.160858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:59.160900 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260929 19:00:59.299386 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:59.299143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:54345)
0929 19:00:59.299265 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:59.299332 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260929 19:01:00.105144 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:00.104912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:57577)
0929 19:01:00.105055 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:00.105096 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260929 19:01:00.161815 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:00.161503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:59985)
0929 19:01:00.161655 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:00.161732 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260929 19:01:00.300088 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:00.299813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:37031)
0929 19:01:00.299948 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:00.299986 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260929 19:01:01.105873 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.105647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:54297)
0929 19:01:01.105782 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.105824 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260929 19:01:01.162509 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.162263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:43613)
0929 19:01:01.162422 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.162464 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260929 19:01:01.300848 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.300623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:54223)
0929 19:01:01.300757 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.300797 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260929 19:01:02.106590 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:02.106366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:39505)
0929 19:01:02.106509 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:02.106544 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260929 19:01:02.163213 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:02.163015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:34717)
0929 19:01:02.163132 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:02.163168 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260929 19:01:02.301635 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:02.301370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:47103)
0929 19:01:02.301542 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:02.301586 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260929 19:01:03.107375 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:03.107087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:58071)
0929 19:01:03.107218 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:03.107264 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260929 19:01:03.163954 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:03.163698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:44989)
0929 19:01:03.163847 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:03.163905 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260929 19:01:03.302400 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:03.302102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:57789)
0929 19:01:03.302315 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:03.302352 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":153,"threads_started":1}
I20260929 19:01:03.604135 15669 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:03.604265 15669 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:01:03.604457 15669 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
W20260929 19:01:03.604861 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
W20260929 19:01:03.604923 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:01:03.604938 15252 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008; no voters: 4d792f9bf1bb4a30a813d832c8f58b50, e509dbb02a6e4ef5916119d8e45f0259
I20260929 19:01:03.605000 15669 raft_consensus.cc:2726] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:01:04.108135 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.107893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:43227)
0929 19:01:04.108039 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:04.108080 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260929 19:01:04.164821 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.164516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:48429)
0929 19:01:04.164719 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:04.164772 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260929 19:01:04.303129 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.302859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:56485)
0929 19:01:04.303037 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:04.303080 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260929 19:01:05.108947 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.108651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:57859)
0929 19:01:05.108792 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:05.108825 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260929 19:01:05.165632 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.165388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:36627)
0929 19:01:05.165539 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:05.165581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260929 19:01:05.303927 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.303692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:34863)
0929 19:01:05.303827 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:05.303870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260929 19:01:05.967391 15676 raft_consensus.cc:482] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:05.967497 15676 raft_consensus.cc:504] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509dbb02a6e4ef5916119d8e45f0259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36499 } } peers { permanent_uuid: "4d792f9bf1bb4a30a813d832c8f58b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } } peers { permanent_uuid: "6d17756bb3ea478b85ec6c316efa1008" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45441 } }
I20260929 19:01:05.967684 15676 leader_election.cc:288] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499), 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329)
W20260929 19:01:05.968053 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e509dbb02a6e4ef5916119d8e45f0259 (127.0.98.3:36499): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
W20260929 19:01:05.968137 15252 leader_election.cc:341] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d792f9bf1bb4a30a813d832c8f58b50 (127.0.98.2:40329): Not found: Tablet not found: 23a8fcd0585f47258f3c37acd9b57fd0
I20260929 19:01:05.968173 15252 leader_election.cc:302] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d17756bb3ea478b85ec6c316efa1008; no voters: 4d792f9bf1bb4a30a813d832c8f58b50, e509dbb02a6e4ef5916119d8e45f0259
I20260929 19:01:05.968255 15676 raft_consensus.cc:2726] T 23a8fcd0585f47258f3c37acd9b57fd0 P 6d17756bb3ea478b85ec6c316efa1008 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:01:06.109751 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.109511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:46547)
0929 19:01:06.109651 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.109692 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260929 19:01:06.166250 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.166034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:59403)
0929 19:01:06.166167 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.166204 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260929 19:01:06.304654 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.304400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:45881)
0929 19:01:06.304540 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.304582 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260929 19:01:07.110469 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:07.110263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:59341)
0929 19:01:07.110384 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:07.110425 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260929 19:01:07.167026 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:07.166823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:52761)
0929 19:01:07.166947 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:07.166980 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260929 19:01:07.305302 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:07.305084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:48937)
0929 19:01:07.305217 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:07.305255 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260929 19:01:08.111203 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.110978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:35639)
0929 19:01:08.111118 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.111150 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260929 19:01:08.167822 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.167577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:34923)
0929 19:01:08.167728 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.167774 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260929 19:01:08.305994 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.305775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:46481)
0929 19:01:08.305898 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.305947 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260929 19:01:09.112100 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.111876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.1:37349)
0929 19:01:09.112013 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.112055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260929 19:01:09.168537 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.168339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.2:55101)
0929 19:01:09.168457 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.168491 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260929 19:01:09.306728 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.306511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.98.3:47489)
0929 19:01:09.306638 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.306676 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260929 19:01:09.307886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15207
I20260929 19:01:09.313086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15370
I20260929 19:01:09.317338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15503
2026-09-29T19:01:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18667 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:09 dist-test-slave-gfsd krb5kdc[15691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:09 dist-test-slave-gfsd krb5kdc[15691](info): set up 2 sockets
Sep 29 19:01:09 dist-test-slave-gfsd krb5kdc[15691](info): commencing operation
krb5kdc: starting...
W20260929 19:01:10.371433 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:10.498656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:10.581116 15704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.581315 15704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.581337 15704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.582394 15704 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:01:10.582425 15704 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:01:10.582437 15704 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:01:10.582449 15704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:10.582461 15704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.582474 15704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.582486 15704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:10.582499 15704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:10.584060 15704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:10.584256 15704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.584421 15704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:10.589205 15710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.589224 15709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.589219 15712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.589636 15704 server_base.cc:1034] running on GCE node
I20260929 19:01:10.589864 15704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.590214 15704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.591347 15704 hybrid_clock.cc:648] HybridClock initialized: now 1790708470591326 us; error 39 us; skew 500 ppm
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:10.593942 15704 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:10.594453 15704 webserver.cc:460] Webserver started at http://127.0.98.62:36687/ using document root <none> and password file <none>
I20260929 19:01:10.594713 15704 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.594801 15704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.594960 15704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.596163 15704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "0e2adc146a124ab7bce1654bee883895"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "73284cb3d717fb1478fc9754ff1ed738"
server_key_iv: "bff101e9c092518a0d00460168cee8d9"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.596536 15704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e2adc146a124ab7bce1654bee883895"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "73284cb3d717fb1478fc9754ff1ed738"
server_key_iv: "bff101e9c092518a0d00460168cee8d9"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.597951 15704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:01:10.598816 15719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.599013 15704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:10.599129 15704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "0e2adc146a124ab7bce1654bee883895"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "73284cb3d717fb1478fc9754ff1ed738"
server_key_iv: "bff101e9c092518a0d00460168cee8d9"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.599210 15704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.614107 15704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.616112 15704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.616231 15704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.620435 15704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41755
I20260929 19:01:10.620467 15771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41755 every 8 connection(s)
I20260929 19:01:10.620829 15704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:10.621670 15772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:10.623485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15704
I20260929 19:01:10.623564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:10.623759 392 external_mini_cluster.cc:1428] Setting key 59026699fd3dd13e52d6bd7ed534fd12
I20260929 19:01:10.624259 15772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap starting.
I20260929 19:01:10.624960 15772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:10.625298 15772 log.cc:824] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:10.625982 15772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: No bootstrap required, opened a new log
I20260929 19:01:10.628327 15772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:10.628487 15772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:10.628518 15772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Initialized, Role: FOLLOWER
I20260929 19:01:10.628633 15772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:10.628706 15772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:10.628734 15772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:10.628779 15772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:10.629513 15772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:10.629631 15772 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2adc146a124ab7bce1654bee883895; no voters:
I20260929 19:01:10.629822 15772 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:10.629915 15777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:10.630139 15777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 LEADER]: Becoming Leader. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Running, Role: LEADER
I20260929 19:01:10.630153 15772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:10.630318 15777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:10.630720 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2adc146a124ab7bce1654bee883895. Latest consensus state: current_term: 1 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:10.630980 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.630899 15778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:10.631531 15778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:10.632319 15788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:10.632828 15788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:10.634385 15788 catalog_manager.cc:1347] Generated new cluster ID: 185b3bc3e0de4a62b98b18ef2efab441
I20260929 19:01:10.634434 15788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:10.635941 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.624284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.98.62:41755)
0929 19:01:10.624442 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.624451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.624482 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:10.625091 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.625102 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.625171 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.625327 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.626067 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.627500 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.628236 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.628832 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.628944 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.632272 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.632276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.632276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:10.632286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.634249 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.634658 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.634661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.634661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.634778 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.635003 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.635004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.635004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.635339 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.635495 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.635590 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.635732 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260929 19:01:10.660691 15788 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:10.661468 15788 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:10.666462 15788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Generated new TSK 0
I20260929 19:01:10.666651 15788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:10.699275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:10.781050 15798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.781292 15798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.781335 15798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.782455 15798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.782521 15798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.782567 15798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:10.784044 15798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:10.784309 15798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.784535 15798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:01:10.789811 15798 server_base.cc:1034] running on GCE node
W20260929 19:01:10.789820 15803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.789810 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.789834 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.790283 15798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.790522 15798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.791663 15798 hybrid_clock.cc:648] HybridClock initialized: now 1790708470791650 us; error 30 us; skew 500 ppm
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:10.794287 15798 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:10.794806 15798 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260929 19:01:10.795032 15798 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.795075 15798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.795218 15798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.796326 15798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "2cbc0bee97ae2c7ff7162fef22961c82"
server_key_iv: "82303bcfe8f371992ac10e8a9836f1be"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.796703 15798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "2cbc0bee97ae2c7ff7162fef22961c82"
server_key_iv: "82303bcfe8f371992ac10e8a9836f1be"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.798050 15798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:10.798856 15813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.799046 15798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:10.799203 15798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "2cbc0bee97ae2c7ff7162fef22961c82"
server_key_iv: "82303bcfe8f371992ac10e8a9836f1be"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.799284 15798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.810184 15798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.812211 15798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.812366 15798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.812597 15798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:10.812901 15798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:10.812954 15798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.812994 15798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:10.813022 15798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.818928 15798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44437
I20260929 19:01:10.818976 15926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44437 every 8 connection(s)
I20260929 19:01:10.819363 15798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:10.824584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15798
I20260929 19:01:10.824698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:10.824898 392 external_mini_cluster.cc:1428] Setting key 069621c4bd840655dd3c05c508bc36a8
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:10.829852 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.820141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45545 (local address 127.0.98.62:41755)
0929 19:01:10.820207 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.820209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.820380 (+ 171us) server_negotiation.cc:407] Connection header received
0929 19:01:10.820855 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.820856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.820865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.820918 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.821878 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.822414 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.823502 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.823726 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.823815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.826339 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.826341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.826341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:10.826348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.828364 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.828761 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.828762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.828763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.828836 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.829096 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.829098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.829098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.829258 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.829340 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.829623 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.829726 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:01:10.829933 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.820072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:45545)
0929 19:01:10.820245 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:10.820287 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:10.820771 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:10.820978 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:10.820982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:10.821543 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:10.821749 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.821754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.822508 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.822509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.823385 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.823388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.823830 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.823830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.823930 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.824470 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:10.824481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:10.826215 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:10.828463 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:10.828470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:10.828471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:10.828682 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:10.828918 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:10.828918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:10.828919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:10.829016 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:10.829352 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:10.829358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:10.829475 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 19:01:10.829584 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:01:10.829709 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260929 19:01:10.830534 15927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:10.830627 15927 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:10.830819 15927 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:10.831435 15736 ts_manager.cc:194] Registered new tserver with Master: 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:10.832108 15736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45545
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:10.853950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:10.938920 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.939144 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.939168 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.940380 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.940424 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.940454 15932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:10.941871 15932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:10.942133 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.942368 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:10.947541 15937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.947536 15938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.947571 15940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.948154 15932 server_base.cc:1034] running on GCE node
I20260929 19:01:10.948341 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.948530 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.949729 15932 hybrid_clock.cc:648] HybridClock initialized: now 1790708470949682 us; error 28 us; skew 500 ppm
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:10.952705 15932 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:10.953085 15932 webserver.cc:460] Webserver started at http://127.0.98.2:34491/ using document root <none> and password file <none>
I20260929 19:01:10.953311 15932 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.953359 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.953508 15932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.954569 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "57eab53ccb264af7bb503929ec36f6f4"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "dd2daa54ac36afd885913ca99feabee4"
server_key_iv: "40cea8abc1a2db5453aeefdd16f51231"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.954936 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "57eab53ccb264af7bb503929ec36f6f4"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "dd2daa54ac36afd885913ca99feabee4"
server_key_iv: "40cea8abc1a2db5453aeefdd16f51231"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.956310 15932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:10.957180 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.957396 15932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:10.957516 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "57eab53ccb264af7bb503929ec36f6f4"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "dd2daa54ac36afd885913ca99feabee4"
server_key_iv: "40cea8abc1a2db5453aeefdd16f51231"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.957585 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.966964 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.969172 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.969341 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.969583 15932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:10.969897 15932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:10.969951 15932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.969985 15932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:10.970011 15932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.975744 15932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37331
I20260929 19:01:10.975834 16060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37331 every 8 connection(s)
I20260929 19:01:10.976123 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:10.978786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 15932
I20260929 19:01:10.978871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:10.979130 392 external_mini_cluster.cc:1428] Setting key f707807e861c85f2afbb1683b5c094ce
Sep 29 19:01:10 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:10.986994 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.976865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51867 (local address 127.0.98.62:41755)
0929 19:01:10.976911 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.976913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.977092 (+ 179us) server_negotiation.cc:407] Connection header received
0929 19:01:10.977573 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.977574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.977582 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.977637 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.978499 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.979107 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.980573 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.980758 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.980819 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.983467 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.983468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.983468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:10.983477 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.985248 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.985667 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.985669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.985669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.985736 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:10.986180 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:10.986181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:10.986181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:10.986328 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.986409 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.986778 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.986871 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:10.987217 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.976830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:51867)
0929 19:01:10.976979 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:10.977020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:10.977492 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:10.977690 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:10.977693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:10.978233 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:10.978414 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.978419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.979198 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.979199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.980487 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.980490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.980839 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.980840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.980985 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.981487 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:10.981495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:10.983373 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:10.985343 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:10.985349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:10.985350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:10.985585 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:10.985843 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:10.985844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:10.985844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:10.986100 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:10.986416 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:10.986423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:10.986599 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 19:01:10.986740 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 19:01:10.986928 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260929 19:01:10.987811 16061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:10.987905 16061 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:10.988098 16061 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:10.988502 15736 ts_manager.cc:194] Registered new tserver with Master: 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 19:01:10.989058 15736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:11.006450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:11.088897 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:11.089103 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:11.089125 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:11.090332 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:11.090376 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:11.090402 16066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:11.091924 16066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:11.092161 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:11.092397 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:11.097556 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:11.097612 16066 server_base.cc:1034] running on GCE node
W20260929 19:01:11.097563 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:11.097512 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:11.097996 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:11.098269 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:11.099402 16066 hybrid_clock.cc:648] HybridClock initialized: now 1790708471099376 us; error 40 us; skew 500 ppm
Sep 29 19:01:11 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:11.102334 16066 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:11.102831 16066 webserver.cc:460] Webserver started at http://127.0.98.3:39193/ using document root <none> and password file <none>
I20260929 19:01:11.103075 16066 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:11.103120 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:11.103266 16066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:11.104311 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
format_stamp: "Formatted at 2026-09-29 19:01:11 on dist-test-slave-gfsd"
server_key: "2effa061b40348cbf71b403b31d0b369"
server_key_iv: "a79eaf0acf9d6da6f6a23a78154eabfa"
server_key_version: "encryptionkey@0"
I20260929 19:01:11.104671 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
format_stamp: "Formatted at 2026-09-29 19:01:11 on dist-test-slave-gfsd"
server_key: "2effa061b40348cbf71b403b31d0b369"
server_key_iv: "a79eaf0acf9d6da6f6a23a78154eabfa"
server_key_version: "encryptionkey@0"
I20260929 19:01:11.106153 16066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:11.106946 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:11.107137 16066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:11.107223 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
format_stamp: "Formatted at 2026-09-29 19:01:11 on dist-test-slave-gfsd"
server_key: "2effa061b40348cbf71b403b31d0b369"
server_key_iv: "a79eaf0acf9d6da6f6a23a78154eabfa"
server_key_version: "encryptionkey@0"
I20260929 19:01:11.107326 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:11.132841 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:11.135205 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:11.135376 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:11.135572 16066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:11.135892 16066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:11.136041 16066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:11.136104 16066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:11.136139 16066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:11.141752 16066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35391
I20260929 19:01:11.141791 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35391 every 8 connection(s)
I20260929 19:01:11.142169 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:01:11 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:11.151590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16066
I20260929 19:01:11.151656 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.142952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38765 (local address 127.0.98.62:41755)
0929 19:01:11.143029 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.143031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.143219 (+ 188us) server_negotiation.cc:407] Connection header received
0929 19:01:11.143786 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.143787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.143797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.143851 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:11.144762 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.145248 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.146199 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.146375 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.146425 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.148794 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:11.148795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:11.148796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:11.148802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:11.150185 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:11.150594 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:11.150595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:11.150595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:11.150650 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:11.150868 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:11.150868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:11.150868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:11.150988 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:11.151044 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.151462 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.151547 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:01:11.151702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:11.151763 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.142913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:38765)
0929 19:01:11.143091 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.143134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.143668 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.143926 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.143930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.144493 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:11.144681 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.144685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.145325 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.145325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.146127 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.146130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.146464 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.146465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.146560 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.147035 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:11.147044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:11.148694 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:11.150265 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:11.150271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:11.150273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:11.150517 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:11.150713 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:11.150713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:11.150714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:11.150813 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:11.151069 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:11.151075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:11.151251 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.151397 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.151550 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260929 19:01:11.151993 392 external_mini_cluster.cc:1428] Setting key 04d58a4b9e2962e1dd316a111bfa9943
I20260929 19:01:11.152223 16195 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:11.152324 16195 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:11.152523 16195 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:11.152977 15736 ts_manager.cc:194] Registered new tserver with Master: 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391)
I20260929 19:01:11.153309 15736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38765
I20260929 19:01:11.163739 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:01:11.171918 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.165211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.0.98.62:41755)
0929 19:01:11.165272 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.165274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.165398 (+ 124us) server_negotiation.cc:407] Connection header received
0929 19:01:11.165437 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.165437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.165445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.165519 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:11.166133 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.166870 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.167798 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.168060 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.168140 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.169588 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:11.169589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:11.169590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:11.169595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:11.170864 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:11.171187 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:11.171188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:11.171188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:11.171239 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:11.171445 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:11.171446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:11.171446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:11.171608 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:11.171692 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.171797 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.171855 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:01:11.173640 15736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:11.175282 15736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:11.182258 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.178001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.0.98.2:37331)
0929 19:01:11.178193 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.178201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.178364 (+ 163us) server_negotiation.cc:407] Connection header received
0929 19:01:11.178515 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.178518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.178535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.178593 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:11.178821 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.180013 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.181328 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.181819 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.181909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.181981 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.182112 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.182164 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260929 19:01:11.182293 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.178392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.98.1:44437)
0929 19:01:11.178648 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.178652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.178661 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:11.178700 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.178701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.178715 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.178787 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:11.178989 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.179780 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.181292 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.181964 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.182037 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.182092 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.182178 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.182215 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260929 19:01:11.182302 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.177865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35391 (local address 127.0.0.1:33816)
0929 19:01:11.178045 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.178087 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.178154 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.178662 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.178665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.178667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:11.178741 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.178742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.180158 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.180159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.181105 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.181107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.182028 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.182029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.182090 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.182094 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.182125 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.182187 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260929 19:01:11.182509 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.177874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37331 (local address 127.0.0.1:34394)
0929 19:01:11.178295 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.178307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.178396 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.178702 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.178703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.178705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:11.178758 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.178760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.180118 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.180119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.181226 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.181228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.181908 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.181909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.181985 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.181999 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.182405 (+ 406us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.182454 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260929 19:01:11.182705 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.178206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.0.98.3:35391)
0929 19:01:11.178383 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.178388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.178460 (+ 72us) server_negotiation.cc:407] Connection header received
0929 19:01:11.178515 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.178517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.178532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.178841 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:11.178925 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.180012 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.181193 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.181918 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.182259 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.182531 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.182577 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.182630 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260929 19:01:11.182749 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.178154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.0.1:45802)
0929 19:01:11.178390 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.178401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.178451 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.178854 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.178857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.178858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:11.178902 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.178903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.179885 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.179887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.181199 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.181202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.182036 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.182037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.182112 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.182116 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.182167 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.182210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260929 19:01:11.183260 15995 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (DEFAULT_TABLE table=test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:11.183341 16129 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (DEFAULT_TABLE table=test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:11.183413 15861 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (DEFAULT_TABLE table=test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:11.183655 15995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:11.183655 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:11.183758 15861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:11.186571 16212 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap starting.
I20260929 19:01:11.187201 16214 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Bootstrap starting.
I20260929 19:01:11.187355 16212 tablet_bootstrap.cc:654] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:11.188120 16214 tablet_bootstrap.cc:654] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:11.188179 16212 log.cc:824] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:11.188581 16213 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Bootstrap starting.
I20260929 19:01:11.188704 16214 log.cc:824] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:11.189013 16212 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: No bootstrap required, opened a new log
I20260929 19:01:11.189106 16212 ts_tablet_manager.cc:1397] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 19:01:11.189288 16213 tablet_bootstrap.cc:654] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:11.189670 16214 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: No bootstrap required, opened a new log
I20260929 19:01:11.189800 16214 ts_tablet_manager.cc:1397] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 19:01:11.189845 16213 log.cc:824] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:11.190639 16213 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: No bootstrap required, opened a new log
I20260929 19:01:11.190711 16213 ts_tablet_manager.cc:1397] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:11.190749 16212 raft_consensus.cc:348] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.190896 16212 raft_consensus.cc:374] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:11.190928 16212 raft_consensus.cc:729] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01926de2c1c840ea9d367d4cd0cadc1b, State: Initialized, Role: FOLLOWER
I20260929 19:01:11.191046 16212 consensus_queue.cc:260] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.191375 15927 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
I20260929 19:01:11.191366 16212 ts_tablet_manager.cc:1428] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:01:11.192302 16214 raft_consensus.cc:348] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.192469 16214 raft_consensus.cc:374] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:11.192562 16214 raft_consensus.cc:729] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f8764387bc4d189c5cd9cfdd5d2135, State: Initialized, Role: FOLLOWER
I20260929 19:01:11.192677 16214 consensus_queue.cc:260] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.192940 16214 ts_tablet_manager.cc:1428] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:01:11.192829 16213 raft_consensus.cc:348] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.192986 16213 raft_consensus.cc:374] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:11.192977 16195 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
I20260929 19:01:11.193029 16213 raft_consensus.cc:729] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57eab53ccb264af7bb503929ec36f6f4, State: Initialized, Role: FOLLOWER
I20260929 19:01:11.193140 16213 consensus_queue.cc:260] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.193436 16061 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
I20260929 19:01:11.193742 16213 ts_tablet_manager.cc:1428] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 19:01:11.226709 16062 tablet.cc:2369] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:11.287261 16219 raft_consensus.cc:482] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:11.287451 16219 raft_consensus.cc:504] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.287858 16219 leader_election.cc:288] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331), 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:11.291006 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.288241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55875 (local address 127.0.98.2:37331)
0929 19:01:11.288299 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.288301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.288401 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:01:11.288495 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.288496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.288522 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.288583 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:11.288750 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.289274 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290265 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.290638 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290731 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.290741 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.290885 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.290922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:01:11.291020 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.288457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32893 (local address 127.0.98.1:44437)
0929 19:01:11.288508 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.288510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.288520 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:11.288604 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.288605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.288616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.288657 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:11.288820 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.289283 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290292 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.290665 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290748 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.290758 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.290911 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.290951 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:01:11.291052 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.288163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37331 (local address 127.0.98.3:55875)
0929 19:01:11.288298 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.288309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.288412 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.288592 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.288593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.288595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:11.288676 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.288678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.289379 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.289380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.290170 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.290172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.290731 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.290809 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.290812 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.290879 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.290921 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1}
I20260929 19:01:11.291064 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.288273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.98.3:32893)
0929 19:01:11.288465 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.288476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:11.288535 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:11.288669 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:11.288670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:11.288671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:11.288738 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.288740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.289368 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.289369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.290198 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:11.290200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.290751 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.290751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:11.290826 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.290829 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:11.290906 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:01:11.290951 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":18,"thread_start_us":116,"threads_started":1}
I20260929 19:01:11.291450 15881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef2cf8a1b9a46ed9385d07e5b61aaac" candidate_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" is_pre_election: true
I20260929 19:01:11.291445 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef2cf8a1b9a46ed9385d07e5b61aaac" candidate_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57eab53ccb264af7bb503929ec36f6f4" is_pre_election: true
I20260929 19:01:11.291607 15881 raft_consensus.cc:2454] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f8764387bc4d189c5cd9cfdd5d2135 in term 0.
I20260929 19:01:11.291607 16015 raft_consensus.cc:2454] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f8764387bc4d189c5cd9cfdd5d2135 in term 0.
I20260929 19:01:11.291862 16082 leader_election.cc:302] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b, 08f8764387bc4d189c5cd9cfdd5d2135; no voters:
I20260929 19:01:11.292048 16219 raft_consensus.cc:2781] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:11.292124 16219 raft_consensus.cc:482] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:11.292143 16219 raft_consensus.cc:3037] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:11.292868 16219 raft_consensus.cc:504] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.292976 16219 leader_election.cc:288] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [CANDIDATE]: Term 1 election: Requested vote from peers 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331), 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:11.293195 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef2cf8a1b9a46ed9385d07e5b61aaac" candidate_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57eab53ccb264af7bb503929ec36f6f4"
I20260929 19:01:11.293272 15881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef2cf8a1b9a46ed9385d07e5b61aaac" candidate_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
I20260929 19:01:11.293303 16015 raft_consensus.cc:3037] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:11.293373 15881 raft_consensus.cc:3037] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:11.294042 16015 raft_consensus.cc:2454] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f8764387bc4d189c5cd9cfdd5d2135 in term 1.
I20260929 19:01:11.294255 16082 leader_election.cc:302] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4; no voters:
I20260929 19:01:11.294319 15881 raft_consensus.cc:2454] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f8764387bc4d189c5cd9cfdd5d2135 in term 1.
I20260929 19:01:11.294366 16219 raft_consensus.cc:2781] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:11.294550 16219 raft_consensus.cc:686] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 LEADER]: Becoming Leader. State: Replica: 08f8764387bc4d189c5cd9cfdd5d2135, State: Running, Role: LEADER
I20260929 19:01:11.294687 16219 consensus_queue.cc:237] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:11.295576 15735 catalog_manager.cc:5495] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } health_report { overall_health: UNKNOWN } } }
W20260929 19:01:11.320143 15928 tablet.cc:2369] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:11.330057 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.327811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.0.98.3:35391)
0929 19:01:11.327873 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:11.327875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:11.327886 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:01:11.327977 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:11.327978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:11.328002 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:11.328066 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:11.328193 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.328657 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.329355 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:11.329550 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:11.329607 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:11.329735 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:11.329832 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:11.329891 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:11.329960 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 19:01:11.329978 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:01:11.331991 16015 raft_consensus.cc:1261] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Refusing update from remote peer 08f8764387bc4d189c5cd9cfdd5d2135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:11.332031 15881 raft_consensus.cc:1261] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Refusing update from remote peer 08f8764387bc4d189c5cd9cfdd5d2135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:11.332407 16222 consensus_queue.cc:1035] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:11.332523 16219 consensus_queue.cc:1035] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:11.337080 15736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:11.337177 15736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:11.337904 15736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Sending DeleteTablet for 3 replicas of tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac
I20260929 19:01:11.338261 15861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45802
I20260929 19:01:11.338296 15995 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34394
I20260929 19:01:11.338510 16235 tablet_replica.cc:330] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: stopping tablet replica
I20260929 19:01:11.338559 16236 tablet_replica.cc:330] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: stopping tablet replica
I20260929 19:01:11.338565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15798
I20260929 19:01:11.338668 16236 raft_consensus.cc:2229] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:11.338753 16235 raft_consensus.cc:2229] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:11.338778 16236 raft_consensus.cc:2258] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:11.338928 16235 raft_consensus.cc:2258] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:11.340307 16129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33816
I20260929 19:01:11.340504 16237 tablet_replica.cc:330] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: stopping tablet replica
I20260929 19:01:11.340585 16237 raft_consensus.cc:2229] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:11.340690 16237 raft_consensus.cc:2258] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:11.341089 16237 ts_tablet_manager.cc:1905] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:11.342804 16237 ts_tablet_manager.cc:1918] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:11.342867 16237 log.cc:1196] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/1ef2cf8a1b9a46ed9385d07e5b61aaac
I20260929 19:01:11.343146 16237 ts_tablet_manager.cc:1939] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting consensus metadata
W20260929 19:01:11.343607 15720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44437 (error 108)
I20260929 19:01:11.343652 15723 catalog_manager.cc:4841] TS 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (table test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]) successfully deleted
I20260929 19:01:11.344173 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.344048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.0.1:45812)
0929 19:01:11.344098 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.344138 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:01:11.344237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15932
W20260929 19:01:11.344282 15720 catalog_manager.cc:4584] TS 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44437: connect: Connection refused (error 111)
I20260929 19:01:11.348996 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:11.348872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37331 (local address 127.0.0.1:34406)
0929 19:01:11.348930 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:11.348959 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:01:11.349167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16066
I20260929 19:01:11.354059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 15704
I20260929 19:01:31.359385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--webserver_interface=127.0.98.62
--webserver_port=36687
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:31.439962 16239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:31.440198 16239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:31.440237 16239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:31.441407 16239 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:01:31.441473 16239 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:01:31.441496 16239 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:01:31.441529 16239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:31.441577 16239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:31.441609 16239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:31.441636 16239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:31.441661 16239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:31.443140 16239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:31.443432 16239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:31.443656 16239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:01:31.452685 16239 server_base.cc:1034] running on GCE node
W20260929 19:01:31.452669 16248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.452869 16246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.452840 16245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:31.453150 16239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:31.453469 16239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:31.454612 16239 hybrid_clock.cc:648] HybridClock initialized: now 1790708491454591 us; error 40 us; skew 500 ppm
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:31.457182 16239 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:31.457629 16239 webserver.cc:460] Webserver started at http://127.0.98.62:36687/ using document root <none> and password file <none>
I20260929 19:01:31.457877 16239 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:31.457942 16239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:31.459321 16239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:31.460017 16255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.460181 16239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:31.460286 16239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "0e2adc146a124ab7bce1654bee883895"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "73284cb3d717fb1478fc9754ff1ed738"
server_key_iv: "bff101e9c092518a0d00460168cee8d9"
server_key_version: "encryptionkey@0"
I20260929 19:01:31.460696 16239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:31.474514 16239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:31.476805 16239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:31.476974 16239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:31.481148 16239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41755
I20260929 19:01:31.481181 16307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41755 every 8 connection(s)
I20260929 19:01:31.481607 16239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:31.483176 16308 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:01:31.483891 16308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap starting.
I20260929 19:01:31.484247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16239
I20260929 19:01:31.484524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44437
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=40025
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:31.492956 16308 log.cc:824] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:31.494879 16308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:31.495177 16308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap complete.
I20260929 19:01:31.497443 16308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:31.497675 16308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.497864 16308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:31.497923 16308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:31.497989 16308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:31.498039 16308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:01:31.499038 16308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:31.499123 16308 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2adc146a124ab7bce1654bee883895; no voters:
I20260929 19:01:31.499323 16308 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:01:31.499396 16312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:01:31.499588 16312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 LEADER]: Becoming Leader. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Running, Role: LEADER
I20260929 19:01:31.499598 16308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:31.499791 16312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:31.500088 16313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:31.500154 16313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:31.500108 16314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2adc146a124ab7bce1654bee883895. Latest consensus state: current_term: 2 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:31.500195 16314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:31.500681 16318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:31.501483 16318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:31.501600 16318 catalog_manager.cc:1259] Loaded cluster ID: 185b3bc3e0de4a62b98b18ef2efab441
I20260929 19:01:31.501833 16318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:31.502633 16318 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:31.502828 16318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Loaded TSK: 0
I20260929 19:01:31.511230 16318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Generated new TSK 1
I20260929 19:01:31.511356 16318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:01:31.583971 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:31.584275 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:31.584328 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:31.585563 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:31.585634 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:31.585682 16310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:31.587216 16310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40025
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:31.587512 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:31.587774 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:31.592835 16338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.592885 16335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.592862 16336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:31.593067 16310 server_base.cc:1034] running on GCE node
I20260929 19:01:31.593281 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:31.593531 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:31.594726 16310 hybrid_clock.cc:648] HybridClock initialized: now 1790708491594699 us; error 45 us; skew 500 ppm
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:31.597203 16310 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:31.597658 16310 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260929 19:01:31.597889 16310 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:31.597972 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:31.599407 16310 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:31.600243 16345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.600401 16310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:31.600508 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "2cbc0bee97ae2c7ff7162fef22961c82"
server_key_iv: "82303bcfe8f371992ac10e8a9836f1be"
server_key_version: "encryptionkey@0"
I20260929 19:01:31.600929 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:31.631961 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:31.634388 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:31.634562 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:31.634816 16310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:31.635267 16352 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:01:31.637193 16310 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:01:31.637261 16310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:01:31.637302 16310 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:01:31.637943 16310 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:01:31.638003 16310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:31.638094 16352 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap starting.
I20260929 19:01:31.645318 16310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44437
I20260929 19:01:31.645463 16459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44437 every 8 connection(s)
I20260929 19:01:31.645740 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:31.649778 16352 log.cc:824] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:31.650496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16310
I20260929 19:01:31.650791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37331
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=34491
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:31.652259 16352 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:31.652587 16352 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap complete.
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:31.653812 16352 ts_tablet_manager.cc:1397] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260929 19:01:31.654999 16352 raft_consensus.cc:348] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:31.655345 16352 raft_consensus.cc:729] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01926de2c1c840ea9d367d4cd0cadc1b, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.655498 16352 consensus_queue.cc:260] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:31.655817 16352 ts_tablet_manager.cc:1428] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260929 19:01:31.657846 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.646455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:54715)
0929 19:01:31.646651 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.646698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.647135 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.647718 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.647722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.648009 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.648183 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.648188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.649389 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.649391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.650465 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.650469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.651482 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.651483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.651635 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.652214 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.652227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.654225 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:31.656258 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.656264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.656266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.656486 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.656760 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.656761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.656761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.656868 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.657245 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:31.657251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:31.657380 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.657480 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.657651 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260929 19:01:31.657847 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.646529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54715 (local address 127.0.98.62:41755)
0929 19:01:31.647345 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.647355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.647399 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:01:31.647513 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.647518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.647557 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.647693 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.648316 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.649245 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.650572 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.651219 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.651370 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.654342 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.654346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.654347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.654359 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.656163 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.656586 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.656589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.656589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.656689 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.656947 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.656948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.656948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.657165 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.657252 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.657544 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.657669 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":100,"threads_started":1}
I20260929 19:01:31.658533 16460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:31.658640 16460 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:31.658835 16460 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:31.659593 16272 ts_manager.cc:194] Registered new tserver with Master: 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:31.660728 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54715
I20260929 19:01:31.661651 16460 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:01:31.668797 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.660385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.98.1:44437)
0929 19:01:31.660539 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.660544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.660553 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:31.660587 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.660589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.660602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.660674 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.661238 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.662178 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.663449 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.663810 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.663884 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.666162 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.666168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.666168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.666174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.667667 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.668077 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.668079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.668079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.668128 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.668319 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.668319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.668319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.668440 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.668498 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.668666 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.668736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260929 19:01:31.668852 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.660150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.0.1:54060)
0929 19:01:31.660327 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.660374 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.660446 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.660704 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.660711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.661088 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.661152 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.661154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.662261 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.662262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.663346 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.663350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.663916 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.663919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.664011 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.664654 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.664658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.666065 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:31.667740 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.667746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.667748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.667998 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.668191 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.668191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.668192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.668256 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.668499 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:31.668505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:31.668554 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.668637 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.668753 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260929 19:01:31.669222 16394 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260929 19:01:31.669771 16469 tablet_replica.cc:330] stopping tablet replica
I20260929 19:01:31.669885 16469 raft_consensus.cc:2229] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:31.669970 16469 raft_consensus.cc:2258] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:31.670235 16469 ts_tablet_manager.cc:1905] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:31.672514 16469 ts_tablet_manager.cc:1918] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:31.672600 16469 log.cc:1196] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal/wals/1ef2cf8a1b9a46ed9385d07e5b61aaac
I20260929 19:01:31.672832 16469 ts_tablet_manager.cc:1939] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 01926de2c1c840ea9d367d4cd0cadc1b: Deleting consensus metadata
I20260929 19:01:31.673305 16256 catalog_manager.cc:4841] TS 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437): tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (table test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]) successfully deleted
W20260929 19:01:31.738435 16465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:31.738619 16465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:31.738641 16465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:31.739825 16465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:31.739866 16465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:31.739892 16465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:31.741277 16465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34491
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:31.741488 16465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:31.741737 16465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:31.746666 16474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.746726 16475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.746636 16477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:31.746894 16465 server_base.cc:1034] running on GCE node
I20260929 19:01:31.747084 16465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:31.747403 16465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:31.748564 16465 hybrid_clock.cc:648] HybridClock initialized: now 1790708491748542 us; error 35 us; skew 500 ppm
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:31.751063 16465 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:31.751521 16465 webserver.cc:460] Webserver started at http://127.0.98.2:34491/ using document root <none> and password file <none>
I20260929 19:01:31.751746 16465 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:31.751804 16465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:31.753113 16465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:31.753774 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.753939 16465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:31.754026 16465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "57eab53ccb264af7bb503929ec36f6f4"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "dd2daa54ac36afd885913ca99feabee4"
server_key_iv: "40cea8abc1a2db5453aeefdd16f51231"
server_key_version: "encryptionkey@0"
I20260929 19:01:31.754453 16465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:31.769497 16465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:31.771871 16465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:31.771984 16465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:31.772164 16465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:31.772552 16491 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:01:31.774049 16465 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:01:31.774097 16465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:01:31.774122 16465 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:01:31.774760 16465 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:01:31.774802 16465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:31.774895 16491 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Bootstrap starting.
I20260929 19:01:31.780550 16465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37331
I20260929 19:01:31.780635 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37331 every 8 connection(s)
I20260929 19:01:31.780911 16465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:31.786209 16491 log.cc:824] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Log is configured to *not* fsync() on all Append() calls
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:31.788655 16491 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:31.788951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16465
I20260929 19:01:31.789005 16491 tablet_bootstrap.cc:492] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Bootstrap complete.
I20260929 19:01:31.789291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35391
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=39193
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:31.790230 16491 ts_tablet_manager.cc:1397] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260929 19:01:31.791531 16491 raft_consensus.cc:348] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:31.791884 16491 raft_consensus.cc:729] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57eab53ccb264af7bb503929ec36f6f4, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.792042 16491 consensus_queue.cc:260] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } }
I20260929 19:01:31.792313 16491 ts_tablet_manager.cc:1428] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260929 19:01:31.792650 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.781709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45949 (local address 127.0.98.62:41755)
0929 19:01:31.781786 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.781789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.781990 (+ 201us) server_negotiation.cc:407] Connection header received
0929 19:01:31.782801 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.782802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.782816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.782862 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.784057 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.784547 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.786242 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.786505 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.786567 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.788957 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.788959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.788959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.788967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.790735 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.791269 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.791270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.791270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.791355 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.791673 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.791674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.791674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.791824 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.791907 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.792511 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.792589 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 19:01:31.792644 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.781603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:45949)
0929 19:01:31.781818 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.781878 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.782616 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.783024 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.783029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.783571 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.783923 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.783933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.784691 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.784692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.786127 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.786130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.786611 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.786612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.786768 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.787410 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.787419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.788852 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:31.790835 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.790841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.790844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.791165 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.791429 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.791430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.791431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.791575 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.791912 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:31.791918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:31.792080 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.792189 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.792374 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260929 19:01:31.793092 16599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:31.793190 16599 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:31.793387 16599 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:31.794004 16272 ts_manager.cc:194] Registered new tserver with Master: 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
I20260929 19:01:31.794607 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45949
I20260929 19:01:31.795552 16599 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 19:01:31.802261 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.794429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.0.98.2:37331)
0929 19:01:31.794583 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.794587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.794852 (+ 265us) server_negotiation.cc:407] Connection header received
0929 19:01:31.794884 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.794886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.794899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.795022 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.795329 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.796331 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.797013 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.797314 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.797394 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.799517 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.799523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.799523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.799528 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.801171 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.801579 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.801581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.801582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.801634 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.801816 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.801816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.801816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.801934 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.801990 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.802130 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.802197 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260929 19:01:31.802306 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.794351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37331 (local address 127.0.0.1:37430)
0929 19:01:31.794729 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.794749 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.794791 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.795026 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.795027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.795185 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.795222 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.795224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.796424 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.796425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.796921 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.796922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.797390 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.797393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.797460 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.797923 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.797928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.799434 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:31.801254 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.801255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.801255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.801497 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.801694 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.801695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.801695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.801757 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.801993 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:31.801994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:31.802037 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.802108 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.802212 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363}
I20260929 19:01:31.802663 16533 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37430
I20260929 19:01:31.803218 16606 tablet_replica.cc:330] stopping tablet replica
I20260929 19:01:31.803390 16606 raft_consensus.cc:2229] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:31.803483 16606 raft_consensus.cc:2258] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:31.803782 16606 ts_tablet_manager.cc:1905] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:31.806133 16606 ts_tablet_manager.cc:1918] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:31.806202 16606 log.cc:1196] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/1ef2cf8a1b9a46ed9385d07e5b61aaac
I20260929 19:01:31.806442 16606 ts_tablet_manager.cc:1939] T 1ef2cf8a1b9a46ed9385d07e5b61aaac P 57eab53ccb264af7bb503929ec36f6f4: Deleting consensus metadata
I20260929 19:01:31.806905 16256 catalog_manager.cc:4841] TS 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): tablet 1ef2cf8a1b9a46ed9385d07e5b61aaac (table test-table [id=c5c394a5983d4ad4b2dfdd9ea39f8af6]) successfully deleted
W20260929 19:01:31.872733 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:31.872922 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:31.872946 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:31.874004 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:31.874039 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:31.874064 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:31.875417 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39193
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:31.875592 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:31.875769 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:31.880740 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:31.880921 16603 server_base.cc:1034] running on GCE node
W20260929 19:01:31.880733 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:31.880856 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:31.881196 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:31.881407 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:31.882624 16603 hybrid_clock.cc:648] HybridClock initialized: now 1790708491882608 us; error 28 us; skew 500 ppm
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:31.885149 16603 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:31.885603 16603 webserver.cc:460] Webserver started at http://127.0.98.3:39193/ using document root <none> and password file <none>
I20260929 19:01:31.885847 16603 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:31.885927 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:31.887264 16603 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:31.887907 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.888069 16603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.888165 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
format_stamp: "Formatted at 2026-09-29 19:01:11 on dist-test-slave-gfsd"
server_key: "2effa061b40348cbf71b403b31d0b369"
server_key_iv: "a79eaf0acf9d6da6f6a23a78154eabfa"
server_key_version: "encryptionkey@0"
I20260929 19:01:31.888556 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:31.896245 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:31.898419 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:31.898583 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:31.898823 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:31.899129 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:31.899183 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.899230 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:31.899276 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:31.905625 16603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35391
I20260929 19:01:31.905735 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35391 every 8 connection(s)
I20260929 19:01:31.906046 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:01:31 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:31.915036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16603
I20260929 19:01:31.915627 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.906783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47735 (local address 127.0.98.62:41755)
0929 19:01:31.906829 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.906831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.907036 (+ 205us) server_negotiation.cc:407] Connection header received
0929 19:01:31.907672 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.907674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.907684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.907786 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.908759 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.909262 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.910370 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.910550 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.910608 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.912812 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.912813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.912814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.912822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.914206 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.914612 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.914613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.914614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.914672 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.914925 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.914926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.914926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.915034 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.915109 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.915452 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.915546 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:31.915884 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.906727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:47735)
0929 19:01:31.906900 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.906943 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.907570 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.907907 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.907911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.908461 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.908648 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.908652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.909338 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.909338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.910247 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.910249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.910626 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.910626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.910731 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.911231 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.911242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.912716 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:31.914305 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.914311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.914313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.914527 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.914741 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:31.914742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:31.914742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:31.914846 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:31.915099 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:31.915105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:31.915265 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.915398 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.915574 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260929 19:01:31.916584 16735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41755
I20260929 19:01:31.916667 16735 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:31.916923 16735 heartbeater.cc:511] Master 127.0.98.62:41755 requested a full tablet report, sending...
I20260929 19:01:31.917330 16272 ts_manager.cc:194] Registered new tserver with Master: 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391)
I20260929 19:01:31.917693 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47735
I20260929 19:01:31.922243 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.915482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.0.98.62:41755)
0929 19:01:31.915629 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.915631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.915834 (+ 203us) server_negotiation.cc:407] Connection header received
0929 19:01:31.915855 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.915855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.915867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.915930 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.916725 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.917368 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.918291 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.918511 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.918595 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.919827 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:31.919828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:31.919829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:31.919835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:31.921083 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.921440 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.921441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.921441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.921500 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:31.921744 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:31.921744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:31.921745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:31.921853 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:31.922016 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.922052 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.922129 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260929 19:01:31.923094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:01:31.925616 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.923489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.0.98.62:41755)
0929 19:01:31.923540 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.923541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.923649 (+ 108us) server_negotiation.cc:407] Connection header received
0929 19:01:31.923694 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.923695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.923717 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.923782 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:31.923845 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.924263 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.924978 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.925156 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.925210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.925386 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:31.925465 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:31.925523 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.925557 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.925574 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:01:31.925864 16272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:31.927438 16272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:31.929438 16394 tablet_service.cc:1467] Processing CreateTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 (DEFAULT_TABLE table=test-table [id=2b1a4668df5a44e48a503361c63a69e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:31.929481 16533 tablet_service.cc:1467] Processing CreateTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 (DEFAULT_TABLE table=test-table [id=2b1a4668df5a44e48a503361c63a69e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:31.929651 16394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fe506a81044182a4d4450b7a20f2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:31.929667 16533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fe506a81044182a4d4450b7a20f2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:31.930914 16491 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Bootstrap starting.
I20260929 19:01:31.931185 16352 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap starting.
I20260929 19:01:31.931258 16491 tablet_bootstrap.cc:654] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:31.931543 16352 tablet_bootstrap.cc:654] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:31.931840 16491 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: No bootstrap required, opened a new log
I20260929 19:01:31.931901 16491 ts_tablet_manager.cc:1397] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:31.932026 16352 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: No bootstrap required, opened a new log
I20260929 19:01:31.932063 16352 ts_tablet_manager.cc:1397] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:31.932196 16491 raft_consensus.cc:348] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.932250 16491 raft_consensus.cc:374] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:31.932277 16491 raft_consensus.cc:729] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57eab53ccb264af7bb503929ec36f6f4, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.932338 16491 consensus_queue.cc:260] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.932444 16491 ts_tablet_manager.cc:1428] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:31.932392 16352 raft_consensus.cc:348] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.932478 16352 raft_consensus.cc:374] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:31.932518 16352 raft_consensus.cc:729] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01926de2c1c840ea9d367d4cd0cadc1b, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.932577 16352 consensus_queue.cc:260] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.932669 16352 ts_tablet_manager.cc:1428] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:31.933809 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.929252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35391 (local address 127.0.0.1:42666)
0929 19:01:31.929349 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.929360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.929395 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.929704 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.929705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.929707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:31.929789 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.929792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.931145 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.931146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.932325 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.932328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.933616 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.933616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.933665 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.933680 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.933735 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.933763 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 19:01:31.933835 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.929314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.3:35391)
0929 19:01:31.929513 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.929518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.929528 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:31.929564 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.929566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.929581 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.929688 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:31.929877 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.930897 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.932568 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.933544 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.933602 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.933640 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.933747 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.933773 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260929 19:01:31.934449 16669 tablet_service.cc:1467] Processing CreateTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 (DEFAULT_TABLE table=test-table [id=2b1a4668df5a44e48a503361c63a69e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:31.934595 16604 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:31.934674 16604 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.934801 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fe506a81044182a4d4450b7a20f2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:31.934976 16604 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:31.938182 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.935110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35391 (local address 127.0.98.2:53183)
0929 19:01:31.935158 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.935168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.935214 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.935418 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.935419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.935420 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:31.935475 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.935477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.936140 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.936141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.937194 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.937196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.937893 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.937894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.937968 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.937990 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.938047 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.938088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:01:31.938179 16745 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Bootstrap starting.
I20260929 19:01:31.938457 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.935163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53183 (local address 127.0.98.3:35391)
0929 19:01:31.935195 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.935197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.935205 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:31.935329 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.935331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.935340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.935538 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:31.935566 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.936063 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.937273 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.937804 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.937882 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.937890 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.938349 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.938395 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:01:31.938879 16745 tablet_bootstrap.cc:654] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:31.938975 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.935440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44463 (local address 127.0.98.1:44437)
0929 19:01:31.935486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.935488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.935495 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:31.935517 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.935518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.935533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.935587 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:31.935766 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.936439 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.937603 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.938125 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.938185 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.938245 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.938872 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.938910 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:01:31.939160 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.935146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.98.2:44463)
0929 19:01:31.935271 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.935316 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.935358 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.935625 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.935625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.935627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:31.935682 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.935683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.936530 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.936531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.937521 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.937523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.938691 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.938692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.938781 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.938784 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:31.938853 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:01:31.938892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":97,"threads_started":1}
I20260929 19:01:31.939241 16745 log.cc:824] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:31.939425 16414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fe506a81044182a4d4450b7a20f2c3" candidate_uuid: "57eab53ccb264af7bb503929ec36f6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" is_pre_election: true
I20260929 19:01:31.939585 16414 raft_consensus.cc:2454] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57eab53ccb264af7bb503929ec36f6f4 in term 0.
I20260929 19:01:31.939852 16485 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b, 57eab53ccb264af7bb503929ec36f6f4; no voters:
I20260929 19:01:31.939967 16604 raft_consensus.cc:2781] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:31.940011 16604 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:31.940040 16604 raft_consensus.cc:3037] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:31.940205 16745 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: No bootstrap required, opened a new log
I20260929 19:01:31.940276 16745 ts_tablet_manager.cc:1397] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:31.938686 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fe506a81044182a4d4450b7a20f2c3" candidate_uuid: "57eab53ccb264af7bb503929ec36f6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" is_pre_election: true
W20260929 19:01:31.940681 16488 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260929 19:01:31.940784 16604 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.940924 16604 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 election: Requested vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437)
I20260929 19:01:31.941062 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fe506a81044182a4d4450b7a20f2c3" candidate_uuid: "57eab53ccb264af7bb503929ec36f6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
I20260929 19:01:31.941084 16414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fe506a81044182a4d4450b7a20f2c3" candidate_uuid: "57eab53ccb264af7bb503929ec36f6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
I20260929 19:01:31.941152 16414 raft_consensus.cc:3037] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 0 FOLLOWER]: Advancing to term 1
W20260929 19:01:31.941241 16488 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260929 19:01:31.941504 16745 raft_consensus.cc:348] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.941650 16745 raft_consensus.cc:374] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:31.941692 16745 raft_consensus.cc:729] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f8764387bc4d189c5cd9cfdd5d2135, State: Initialized, Role: FOLLOWER
I20260929 19:01:31.941761 16414 raft_consensus.cc:2454] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57eab53ccb264af7bb503929ec36f6f4 in term 1.
I20260929 19:01:31.941814 16745 consensus_queue.cc:260] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.941949 16485 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b, 57eab53ccb264af7bb503929ec36f6f4; no voters: 08f8764387bc4d189c5cd9cfdd5d2135
I20260929 19:01:31.942073 16604 raft_consensus.cc:2781] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:31.942091 16745 ts_tablet_manager.cc:1428] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:31.942140 16735 heartbeater.cc:503] Master 127.0.98.62:41755 was elected leader, sending a full tablet report...
I20260929 19:01:31.942214 16604 raft_consensus.cc:686] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 1 LEADER]: Becoming Leader. State: Replica: 57eab53ccb264af7bb503929ec36f6f4, State: Running, Role: LEADER
I20260929 19:01:31.942277 16604 consensus_queue.cc:237] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:31.942953 16271 catalog_manager.cc:5495] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "57eab53ccb264af7bb503929ec36f6f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } health_report { overall_health: HEALTHY } } }
I20260929 19:01:31.950685 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.948330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.0.98.2:37331)
0929 19:01:31.948397 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.948399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.948424 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:01:31.948459 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.948459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.948473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.948534 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:31.948682 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.949193 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.949909 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.950099 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.950156 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.950296 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:31.950379 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:31.950435 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:31.950545 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 19:01:31.950564 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:01:31.951924 16689 raft_consensus.cc:3037] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:31.952014 16414 raft_consensus.cc:1261] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Refusing update from remote peer 57eab53ccb264af7bb503929ec36f6f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:31.952318 16604 consensus_queue.cc:1035] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:31.952891 16689 raft_consensus.cc:1261] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Refusing update from remote peer 57eab53ccb264af7bb503929ec36f6f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:31.953543 16604 consensus_queue.cc:1035] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:31.954543 16752 mvcc.cc:205] Tried to move back new op lower bound from 7334741983033245696 to 7334741982996152320. Current Snapshot: MvccSnapshot[applied={T|T < 7334741983033245696 or (T in {7334741983033245696})}]
I20260929 19:01:31.954856 16751 mvcc.cc:205] Tried to move back new op lower bound from 7334741983033245696 to 7334741982996152320. Current Snapshot: MvccSnapshot[applied={T|T < 7334741983033245696}]
I20260929 19:01:31.956785 16272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:31.956885 16272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:31.957661 16272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Sending DeleteTablet for 3 replicas of tablet f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:31.957950 16533 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37430
I20260929 19:01:31.957962 16394 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260929 19:01:31.957993 16669 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fe506a81044182a4d4450b7a20f2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42666
I20260929 19:01:31.958060 16469 tablet_replica.cc:330] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: stopping tablet replica
I20260929 19:01:31.958124 16469 raft_consensus.cc:2229] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:31.958159 16606 tablet_replica.cc:330] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: stopping tablet replica
I20260929 19:01:31.958182 16469 raft_consensus.cc:2258] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:31.958220 16606 raft_consensus.cc:2229] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:31.958189 16761 tablet_replica.cc:330] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: stopping tablet replica
I20260929 19:01:31.958292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16310
I20260929 19:01:31.958300 16606 raft_consensus.cc:2258] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:31.958309 16761 raft_consensus.cc:2229] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:31.958405 16761 raft_consensus.cc:2258] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:31.959093 16606 ts_tablet_manager.cc:1905] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:31.960428 16761 ts_tablet_manager.cc:1905] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:31.960712 16606 ts_tablet_manager.cc:1918] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:31.960856 16606 log.cc:1196] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:31.961066 16606 ts_tablet_manager.cc:1939] T f9fe506a81044182a4d4450b7a20f2c3 P 57eab53ccb264af7bb503929ec36f6f4: Deleting consensus metadata
I20260929 19:01:31.961388 16256 catalog_manager.cc:4841] TS 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): tablet f9fe506a81044182a4d4450b7a20f2c3 (table test-table [id=2b1a4668df5a44e48a503361c63a69e2]) successfully deleted
I20260929 19:01:31.962411 16761 ts_tablet_manager.cc:1918] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:31.962484 16761 log.cc:1196] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:31.962733 16761 ts_tablet_manager.cc:1939] T f9fe506a81044182a4d4450b7a20f2c3 P 08f8764387bc4d189c5cd9cfdd5d2135: Deleting consensus metadata
I20260929 19:01:31.963321 16259 catalog_manager.cc:4841] TS 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): tablet f9fe506a81044182a4d4450b7a20f2c3 (table test-table [id=2b1a4668df5a44e48a503361c63a69e2]) successfully deleted
W20260929 19:01:31.965040 16256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44437 (error 108)
I20260929 19:01:31.965461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16465
I20260929 19:01:31.965492 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.965352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44437 (local address 127.0.0.1:54062)
0929 19:01:31.965410 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.965455 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:01:31.966207 16256 catalog_manager.cc:4584] TS 01926de2c1c840ea9d367d4cd0cadc1b (127.0.98.1:44437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9fe506a81044182a4d4450b7a20f2c3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44437: connect: Connection refused (error 111)
I20260929 19:01:31.971158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16603
I20260929 19:01:31.975991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16239
I20260929 19:01:51.981671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--webserver_interface=127.0.98.62
--webserver_port=36687
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:52.064821 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:52.065151 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:52.065192 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:52.066339 16763 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:01:52.066411 16763 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:01:52.066437 16763 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:01:52.066458 16763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:52.066490 16763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:52.066529 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:52.066561 16763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:52.066593 16763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:52.068094 16763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:52.068377 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:52.068614 16763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:52.074225 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.074362 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.074369 16763 server_base.cc:1034] running on GCE node
W20260929 19:01:52.074270 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.074858 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:52.075145 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:52.076310 16763 hybrid_clock.cc:648] HybridClock initialized: now 1790708512076290 us; error 44 us; skew 500 ppm
Sep 29 19:01:52 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:52.085430 16763 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:52.085938 16763 webserver.cc:460] Webserver started at http://127.0.98.62:36687/ using document root <none> and password file <none>
I20260929 19:01:52.086172 16763 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:52.086243 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:52.087729 16763 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:52.088455 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.088711 16763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:52.088816 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "0e2adc146a124ab7bce1654bee883895"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "73284cb3d717fb1478fc9754ff1ed738"
server_key_iv: "bff101e9c092518a0d00460168cee8d9"
server_key_version: "encryptionkey@0"
I20260929 19:01:52.089160 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:52.109230 16763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:52.111327 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:52.111569 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:52.115499 16763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41755
I20260929 19:01:52.115530 16830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41755 every 8 connection(s)
I20260929 19:01:52.115922 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:52.117117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16763
I20260929 19:01:52.117424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44437
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=40025
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:52.117756 16831 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:01:52.118917 16831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap starting.
I20260929 19:01:52.127610 16831 log.cc:824] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:52.129894 16831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:52.130218 16831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Bootstrap complete.
I20260929 19:01:52.132560 16831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:52.132778 16831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Initialized, Role: FOLLOWER
I20260929 19:01:52.132988 16831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:52.133075 16831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:52.133137 16831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:52.133191 16831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:01:52.134169 16831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:52.134256 16831 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2adc146a124ab7bce1654bee883895; no voters:
I20260929 19:01:52.134501 16831 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:01:52.134545 16835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:01:52.134714 16835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [term 3 LEADER]: Becoming Leader. State: Replica: 0e2adc146a124ab7bce1654bee883895, State: Running, Role: LEADER
I20260929 19:01:52.134781 16831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:52.134865 16835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } }
I20260929 19:01:52.135151 16835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2adc146a124ab7bce1654bee883895. Latest consensus state: current_term: 3 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:52.135249 16835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:52.135154 16836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0e2adc146a124ab7bce1654bee883895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2adc146a124ab7bce1654bee883895" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41755 } } }
I20260929 19:01:52.135442 16836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:52.135484 16843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:52.136373 16843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:52.136479 16843 catalog_manager.cc:1259] Loaded cluster ID: 185b3bc3e0de4a62b98b18ef2efab441
I20260929 19:01:52.136516 16843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:52.137717 16843 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:52.137923 16843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Loaded TSK: 0
I20260929 19:01:52.137970 16843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e2adc146a124ab7bce1654bee883895: Loaded TSK: 1
F20260929 19:01:52.138192 16843 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2822ce09ad google::LogMessage::Fail() at ??:0
@ 0x7f2822ce388a google::LogMessage::SendToLog() at ??:0
@ 0x7f2822ce04de google::LogMessage::Flush() at ??:0
@ 0x7f2822ce1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f282544d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f282544ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f282ac3a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f282ac3a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f282ac3a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f282ac00c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f282ac0aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2824bae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2824ba3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2825a076db start_thread at ??:0
@ 0x7f28217de61f clone at ??:0
W20260929 19:01:52.225407 16833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:52.225684 16833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:52.225723 16833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:52.227468 16833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:52.227524 16833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:52.227564 16833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:52.229784 16833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40025
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:52.230054 16833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:52.230314 16833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:52.235836 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.236018 16833 server_base.cc:1034] running on GCE node
W20260929 19:01:52.235852 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.235839 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.236406 16833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:52.236682 16833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:52.237859 16833 hybrid_clock.cc:648] HybridClock initialized: now 1790708512237844 us; error 35 us; skew 500 ppm
Sep 29 19:01:52 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:52.240780 16833 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:52.241283 16833 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260929 19:01:52.241583 16833 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:52.241660 16833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:52.243057 16833 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:01:52.243820 16882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.244117 16833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:52.244256 16833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "01926de2c1c840ea9d367d4cd0cadc1b"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "2cbc0bee97ae2c7ff7162fef22961c82"
server_key_iv: "82303bcfe8f371992ac10e8a9836f1be"
server_key_version: "encryptionkey@0"
I20260929 19:01:52.244693 16833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:52.258070 16833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:52.260399 16833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:52.260660 16833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:52.260900 16833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:52.261468 16889 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:01:52.263619 16833 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:01:52.263703 16833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:01:52.263789 16833 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:01:52.264503 16833 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:01:52.264565 16833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:52.264636 16889 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap starting.
I20260929 19:01:52.271912 16833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44437
I20260929 19:01:52.271982 16996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44437 every 8 connection(s)
I20260929 19:01:52.272315 16833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:52.273195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 16833
I20260929 19:01:52.273481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37331
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=34491
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:52.275981 16889 log.cc:824] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:52.278136 16889 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:52.278404 16889 tablet_bootstrap.cc:492] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Bootstrap complete.
I20260929 19:01:52.279363 16889 ts_tablet_manager.cc:1397] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260929 19:01:52.280350 16889 raft_consensus.cc:348] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:52.280591 16889 raft_consensus.cc:729] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01926de2c1c840ea9d367d4cd0cadc1b, State: Initialized, Role: FOLLOWER
I20260929 19:01:52.280752 16889 consensus_queue.cc:260] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:52.281056 16889 ts_tablet_manager.cc:1428] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260929 19:01:52.365823 16999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:52.272999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:41549)
0929 19:01:52.273201 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.273250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:52.273974 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:52.365620 (+ 91646us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
W20260929 19:01:52.366204 16997 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41755: Client connection negotiation failed: client connection to 127.0.98.62:41755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:01:52.366739 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.366594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:38555)
0929 19:01:52.366658 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.366709 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 19:01:52.367154 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.367050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:37139)
0929 19:01:52.367100 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.367130 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:01:52.367326 16997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:01:52.370836 17000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:52.371021 17000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:52.371044 17000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:52.372277 17000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:52.372318 17000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:52.372344 17000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:52.373816 17000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34491
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:52.374066 17000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:52.374298 17000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:52.379352 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.379352 17008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.379380 17007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.379700 17000 server_base.cc:1034] running on GCE node
I20260929 19:01:52.379906 17000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:52.380136 17000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:52.381285 17000 hybrid_clock.cc:648] HybridClock initialized: now 1790708512381263 us; error 35 us; skew 500 ppm
Sep 29 19:01:52 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:52.383955 17000 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:52.384338 17000 webserver.cc:460] Webserver started at http://127.0.98.2:34491/ using document root <none> and password file <none>
I20260929 19:01:52.384526 17000 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:52.384567 17000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:52.385804 17000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:52.386359 17017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.386526 17000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.386615 17000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "57eab53ccb264af7bb503929ec36f6f4"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-gfsd"
server_key: "dd2daa54ac36afd885913ca99feabee4"
server_key_iv: "40cea8abc1a2db5453aeefdd16f51231"
server_key_version: "encryptionkey@0"
I20260929 19:01:52.386914 17000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:52.404819 17000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:52.406991 17000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:52.407102 17000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:52.407354 17000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:52.407693 17000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:52.407733 17000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.407757 17000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:52.407769 17000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.413409 17000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37331
I20260929 19:01:52.413470 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37331 every 8 connection(s)
I20260929 19:01:52.413786 17000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:52.414943 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.414542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:33545)
0929 19:01:52.414733 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.414808 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20260929 19:01:52.415436 17131 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41755: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
I20260929 19:01:52.415742 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.415633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:40687)
0929 19:01:52.415698 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.415724 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 19:01:52.416064 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.415974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:34797)
0929 19:01:52.416018 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.416040 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:01:52.416162 17131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:01:52.419783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17000
I20260929 19:01:52.420017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35391
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41755
--webserver_port=39193
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:52.501883 17134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:52.502116 17134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:52.502154 17134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:52.503350 17134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:52.503415 17134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:52.503461 17134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:52.504909 17134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39193
--tserver_master_addrs=127.0.98.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:01:52.505167 17134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:52.505399 17134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:52.510629 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.510635 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:52.510623 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:52.510876 17134 server_base.cc:1034] running on GCE node
I20260929 19:01:52.511132 17134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:52.511445 17134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:52.512605 17134 hybrid_clock.cc:648] HybridClock initialized: now 1790708512512600 us; error 32 us; skew 500 ppm
Sep 29 19:01:52 dist-test-slave-gfsd krb5kdc[15691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:52.515134 17134 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:52.515677 17134 webserver.cc:460] Webserver started at http://127.0.98.3:39193/ using document root <none> and password file <none>
I20260929 19:01:52.515919 17134 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:52.515990 17134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:52.517271 17134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:52.517920 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.518106 17134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:52.518231 17134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "08f8764387bc4d189c5cd9cfdd5d2135"
format_stamp: "Formatted at 2026-09-29 19:01:11 on dist-test-slave-gfsd"
server_key: "2effa061b40348cbf71b403b31d0b369"
server_key_iv: "a79eaf0acf9d6da6f6a23a78154eabfa"
server_key_version: "encryptionkey@0"
I20260929 19:01:52.518558 17134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:52.548364 17134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:52.550820 17134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:52.550995 17134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:52.551267 17134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:52.551601 17134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:52.551656 17134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.551700 17134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:52.551728 17134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:52.557643 17134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35391
I20260929 19:01:52.557703 17262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35391 every 8 connection(s)
I20260929 19:01:52.558080 17134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:52.559149 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.558791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:51947)
0929 19:01:52.558944 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.559010 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
W20260929 19:01:52.559615 17263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41755: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
I20260929 19:01:52.559938 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.559818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:40433)
0929 19:01:52.559888 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.559918 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 19:01:52.560281 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:52.560199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:56931)
0929 19:01:52.560241 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:52.560265 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:01:52.560407 17263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:01:52.565435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17134
I20260929 19:01:53.368045 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.367810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:42469)
0929 19:01:53.367941 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:53.367980 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260929 19:01:53.416769 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.416542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:53503)
0929 19:01:53.416681 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:53.416718 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260929 19:01:53.513430 17269 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:53.513624 17269 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:53.513943 17269 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
Sep 29 19:01:53 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 19:01:53 dist-test-slave-gfsd krb5kdc[15691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 19:01:53.525156 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.514154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35391 (local address 127.0.98.1:44229)
0929 19:01:53.514197 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:53.514209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:53.514268 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:53.515194 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:53.515197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:53.515755 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:53.515938 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:53.515946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.517249 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.517250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:53.518039 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:53.518041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.518641 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:53.518772 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:53.519528 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:53.519532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:53.521654 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:53.523720 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:53.523721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:53.523721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:53.523963 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:53.524234 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:53.524235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:53.524236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:53.524297 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:53.524694 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:53.524694 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:53.524743 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 19:01:53.524847 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 19:01:53.525059 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":40}
I20260929 19:01:53.525173 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.514311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32993 (local address 127.0.98.2:37331)
0929 19:01:53.514466 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:53.514474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:53.514503 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:01:53.514923 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:53.514925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:53.514969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:53.515147 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:53.516169 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.517098 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518151 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.518561 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518696 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:53.521168 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:53.521171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:53.521172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:53.521182 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:53.523164 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:53.523728 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:53.523731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:53.523732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:53.523839 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:53.524131 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:53.524132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:53.524132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:53.524523 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:53.524587 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:53.524916 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 19:01:53.525049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":49,"threads_started":1}
I20260929 19:01:53.525218 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.514282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44229 (local address 127.0.98.3:35391)
0929 19:01:53.514422 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:53.514435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:53.514468 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:01:53.514884 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:53.514886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:53.514936 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:53.515561 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:53.516210 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.517121 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518150 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.518545 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518667 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:53.521743 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:53.521750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:53.521750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:53.521760 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:53.523627 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:53.524055 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:53.524058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:53.524059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:53.524158 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:53.524364 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:53.524364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:53.524364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:53.524611 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:53.524692 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:53.524914 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 19:01:53.525102 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260929 19:01:53.525722 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.514154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37331 (local address 127.0.98.1:32993)
0929 19:01:53.514315 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:53.514323 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:53.514364 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:53.515359 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:53.515360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:53.515753 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:53.515939 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:53.515945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.517202 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.517204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:53.518041 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:53.518043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:53.518671 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:53.518672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:53.518772 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:53.519457 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:53.519465 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:53.521043 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:53.523250 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:53.523257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:53.523259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:53.523591 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:53.523917 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:53.523918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:53.523918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:53.524052 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:53.524597 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:53.524604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:53.524741 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 19:01:53.524847 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:01:53.525143 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":279,"thread_start_us":49,"threads_started":1}
W20260929 19:01:53.526366 16886 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
W20260929 19:01:53.526579 16883 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:53.526623 16883 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b; no voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4
I20260929 19:01:53.526739 17269 raft_consensus.cc:2726] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:01:53.560976 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:53.560719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:36799)
0929 19:01:53.560886 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:53.560939 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260929 19:01:54.368945 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:54.368662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:58363)
0929 19:01:54.368822 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:54.368893 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260929 19:01:54.417462 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:54.417261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:34187)
0929 19:01:54.417381 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:54.417420 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260929 19:01:54.561754 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:54.561482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:41107)
0929 19:01:54.561668 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:54.561702 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260929 19:01:55.181816 17277 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:55.181926 17277 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:55.182097 17277 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
W20260929 19:01:55.182548 16886 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
W20260929 19:01:55.182626 16883 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:55.182704 16886 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b; no voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4
I20260929 19:01:55.182811 17277 raft_consensus.cc:2726] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:01:55.369707 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:55.369487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:53409)
0929 19:01:55.369615 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:55.369657 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260929 19:01:55.418171 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:55.417979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:59433)
0929 19:01:55.418095 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:55.418129 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260929 19:01:55.562523 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:55.562268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:54147)
0929 19:01:55.562438 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:55.562481 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260929 19:01:56.370568 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:56.370320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:34627)
0929 19:01:56.370477 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:56.370519 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260929 19:01:56.418895 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:56.418665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:48587)
0929 19:01:56.418792 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:56.418838 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260929 19:01:56.563266 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:56.563020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:35199)
0929 19:01:56.563171 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:56.563219 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260929 19:01:57.371414 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:57.371149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:60613)
0929 19:01:57.371266 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:57.371367 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260929 19:01:57.419715 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:57.419490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:54123)
0929 19:01:57.419618 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:57.419665 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260929 19:01:57.564080 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:57.563856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:58567)
0929 19:01:57.563996 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:57.564035 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260929 19:01:57.659796 17287 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:57.659904 17287 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:01:57.660101 17287 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
W20260929 19:01:57.660754 16886 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
W20260929 19:01:57.660859 16883 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:01:57.660913 16883 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b; no voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4
I20260929 19:01:57.660987 17287 raft_consensus.cc:2726] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:01:58.372236 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:58.372000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:38199)
0929 19:01:58.372142 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:58.372186 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260929 19:01:58.420496 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:58.420267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:44101)
0929 19:01:58.420405 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:58.420443 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260929 19:01:58.564738 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:58.564511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:34425)
0929 19:01:58.564647 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:58.564683 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260929 19:01:59.372995 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:59.372762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:48677)
0929 19:01:59.372903 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:59.372943 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260929 19:01:59.421324 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:59.421007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:60119)
0929 19:01:59.421201 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:59.421261 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":130,"threads_started":1}
I20260929 19:01:59.565389 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:59.565158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:57785)
0929 19:01:59.565310 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:59.565346 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260929 19:02:00.373762 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:00.373493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:57463)
0929 19:02:00.373676 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:00.373715 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":61,"threads_started":1}
I20260929 19:02:00.422035 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:00.421811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:39417)
0929 19:02:00.421944 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:00.421990 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260929 19:02:00.566061 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:00.565855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:37717)
0929 19:02:00.565974 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:00.566016 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260929 19:02:00.607853 17297 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:00.607947 17297 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:02:00.608228 17297 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
W20260929 19:02:00.608666 16886 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
W20260929 19:02:00.608757 16883 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:02:00.608834 16883 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b; no voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4
I20260929 19:02:00.608920 17297 raft_consensus.cc:2726] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:02:01.374449 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:01.374241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:36725)
0929 19:02:01.374365 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:01.374399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260929 19:02:01.422854 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:01.422606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:43235)
0929 19:02:01.422767 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:01.422809 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260929 19:02:01.566779 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:01.566539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:41071)
0929 19:02:01.566695 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:01.566734 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260929 19:02:02.375178 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:02.374953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:46675)
0929 19:02:02.375096 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:02.375129 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260929 19:02:02.423588 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:02.423374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:58083)
0929 19:02:02.423500 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:02.423540 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260929 19:02:02.567548 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:02.567326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:38115)
0929 19:02:02.567467 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:02.567504 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260929 19:02:03.375898 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:03.375694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:40889)
0929 19:02:03.375815 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:03.375852 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 19:02:03.424278 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:03.424046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:37343)
0929 19:02:03.424193 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:03.424233 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260929 19:02:03.568466 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:03.568201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:59509)
0929 19:02:03.568379 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:03.568420 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260929 19:02:04.376681 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:04.376446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:38401)
0929 19:02:04.376598 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:04.376634 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260929 19:02:04.425004 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:04.424771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:44099)
0929 19:02:04.424915 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:04.424957 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260929 19:02:04.569234 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:04.569001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:42851)
0929 19:02:04.569150 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:04.569189 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":121,"threads_started":1}
I20260929 19:02:05.113452 17310 raft_consensus.cc:482] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:05.113557 17310 raft_consensus.cc:504] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f8764387bc4d189c5cd9cfdd5d2135" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35391 } } peers { permanent_uuid: "01926de2c1c840ea9d367d4cd0cadc1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44437 } } peers { permanent_uuid: "57eab53ccb264af7bb503929ec36f6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37331 } }
I20260929 19:02:05.113822 17310 leader_election.cc:288] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391), 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331)
W20260929 19:02:05.114284 16883 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57eab53ccb264af7bb503929ec36f6f4 (127.0.98.2:37331): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
W20260929 19:02:05.114377 16886 leader_election.cc:341] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08f8764387bc4d189c5cd9cfdd5d2135 (127.0.98.3:35391): Not found: Tablet not found: f9fe506a81044182a4d4450b7a20f2c3
I20260929 19:02:05.114585 16886 leader_election.cc:302] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01926de2c1c840ea9d367d4cd0cadc1b; no voters: 08f8764387bc4d189c5cd9cfdd5d2135, 57eab53ccb264af7bb503929ec36f6f4
I20260929 19:02:05.114686 17310 raft_consensus.cc:2726] T f9fe506a81044182a4d4450b7a20f2c3 P 01926de2c1c840ea9d367d4cd0cadc1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:02:05.377417 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:05.377201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:44599)
0929 19:02:05.377333 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:05.377369 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260929 19:02:05.425879 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:05.425633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:36433)
0929 19:02:05.425786 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:05.425830 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260929 19:02:05.569957 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:05.569752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:52723)
0929 19:02:05.569871 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:05.569911 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260929 19:02:06.378335 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:06.378092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:52381)
0929 19:02:06.378239 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:06.378283 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260929 19:02:06.426687 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:06.426432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:45785)
0929 19:02:06.426590 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:06.426635 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260929 19:02:06.570842 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:06.570563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:46771)
0929 19:02:06.570752 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:06.570794 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260929 19:02:07.379101 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:07.378869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.1:58907)
0929 19:02:07.379024 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:07.379055 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260929 19:02:07.427632 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:07.427385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.2:34607)
0929 19:02:07.427545 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:07.427584 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260929 19:02:07.566218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 16833
I20260929 19:02:07.571364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17000
I20260929 19:02:07.572018 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:07.571437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41755 (local address 127.0.98.3:49659)
0929 19:02:07.571928 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:07.571966 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":75,"threads_started":1}
I20260929 19:02:07.576406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17134
2026-09-29T19:02:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58258 ms)
[----------] 30 tests from SecurityITest (233615 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:07 dist-test-slave-gfsd krb5kdc[17322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:07 dist-test-slave-gfsd krb5kdc[17322](info): set up 2 sockets
Sep 29 19:02:07 dist-test-slave-gfsd krb5kdc[17322](info): commencing operation
krb5kdc: starting...
W20260929 19:02:08.614962 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:08 dist-test-slave-gfsd krb5kdc[17322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:08.733619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:08.819231 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:08.819483 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:08.819522 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:08.820694 17335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:08.820772 17335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:08.820799 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:08.820828 17335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:08.820858 17335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:08.822363 17335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:08.822649 17335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:08.822875 17335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:08.827860 17343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:08.827821 17340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:08.827989 17335 server_base.cc:1034] running on GCE node
W20260929 19:02:08.827838 17341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:08.828446 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:08.828732 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:08.829896 17335 hybrid_clock.cc:648] HybridClock initialized: now 1790708528829884 us; error 49 us; skew 500 ppm
Sep 29 19:02:08 dist-test-slave-gfsd krb5kdc[17322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:08.832460 17335 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:08.832875 17335 webserver.cc:460] Webserver started at http://127.0.98.62:42371/ using document root <none> and password file <none>
I20260929 19:02:08.833117 17335 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:08.833189 17335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:08.833333 17335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:08.834375 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54"
format_stamp: "Formatted at 2026-09-29 19:02:08 on dist-test-slave-gfsd"
server_key: "94741d1cc8bd11fe874b1ebc1a12ee1d"
server_key_iv: "8c60251efdb856612652ec06cf423cb3"
server_key_version: "encryptionkey@0"
I20260929 19:02:08.834728 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54"
format_stamp: "Formatted at 2026-09-29 19:02:08 on dist-test-slave-gfsd"
server_key: "94741d1cc8bd11fe874b1ebc1a12ee1d"
server_key_iv: "8c60251efdb856612652ec06cf423cb3"
server_key_version: "encryptionkey@0"
I20260929 19:02:08.836158 17335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:08.836961 17350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:08.837174 17335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:08.837280 17335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54"
format_stamp: "Formatted at 2026-09-29 19:02:08 on dist-test-slave-gfsd"
server_key: "94741d1cc8bd11fe874b1ebc1a12ee1d"
server_key_iv: "8c60251efdb856612652ec06cf423cb3"
server_key_version: "encryptionkey@0"
I20260929 19:02:08.837352 17335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:08.845597 17335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:08.847851 17335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:08.848016 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:08.852214 17335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43427
I20260929 19:02:08.852250 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43427 every 8 connection(s)
I20260929 19:02:08.852609 17335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:08.853549 17403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:08.856148 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: Bootstrap starting.
I20260929 19:02:08.856738 17403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:08.857026 17403 log.cc:824] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:08.857756 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: No bootstrap required, opened a new log
I20260929 19:02:08.858198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17335
I20260929 19:02:08.858319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:08.858530 392 external_mini_cluster.cc:1428] Setting key be5e3736e2973bd4ad6134963038c437
I20260929 19:02:08.859972 17403 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260929 19:02:08.860129 17403 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:08.860193 17403 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d5b7b1b86e48f5bbba5e92e25d6d54, State: Initialized, Role: FOLLOWER
I20260929 19:02:08.860378 17403 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260929 19:02:08.860474 17403 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:08.860518 17403 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:08.860566 17403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:08.861411 17403 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260929 19:02:08.861531 17403 leader_election.cc:302] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0d5b7b1b86e48f5bbba5e92e25d6d54; no voters:
I20260929 19:02:08.861704 17403 leader_election.cc:288] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:08.861750 17408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:08.861918 17408 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [term 1 LEADER]: Becoming Leader. State: Replica: a0d5b7b1b86e48f5bbba5e92e25d6d54, State: Running, Role: LEADER
I20260929 19:02:08.862066 17408 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260929 19:02:08.862106 17403 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:08.862438 17410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d5b7b1b86e48f5bbba5e92e25d6d54. Latest consensus state: current_term: 1 leader_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } } }
I20260929 19:02:08.862489 17409 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d5b7b1b86e48f5bbba5e92e25d6d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } } }
I20260929 19:02:08.862758 17410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:08.863044 17409 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:08.863229 17414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:08.863822 17414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:02:08 dist-test-slave-gfsd krb5kdc[17322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:08.866592 17414 catalog_manager.cc:1347] Generated new cluster ID: b318f02a3b8b44eeb3bc70aacc56fead
I20260929 19:02:08.866652 17414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:08.870350 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:08.859047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.0.98.62:43427)
0929 19:02:08.859258 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:08.859268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:08.859328 (+ 60us) server_negotiation.cc:407] Connection header received
0929 19:02:08.859955 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:08.859961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:08.860007 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:08.860151 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:08.860889 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:08.861978 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:08.863377 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:08.864075 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:08.864221 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:08.866778 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:08.866782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:08.866782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:08.866794 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:08.868610 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:08.868989 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:08.868991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:08.868992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:08.869111 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:08.869328 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:08.869328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:08.869329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:08.869562 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:08.869715 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:08.869811 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 19:02:08.870050 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260929 19:02:08.876771 17414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:08.877476 17414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:08.896334 17414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: Generated new TSK 0
I20260929 19:02:08.896559 17414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:02:08.929880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:09.048231 17429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.048503 17429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.048538 17429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.050099 17429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.050158 17429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.050199 17429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:09.051959 17429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.052155 17429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.052330 17429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.057511 17434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.057591 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.057638 17429 server_base.cc:1034] running on GCE node
W20260929 19:02:09.057510 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.058135 17429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.058353 17429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.059499 17429 hybrid_clock.cc:648] HybridClock initialized: now 1790708529059481 us; error 44 us; skew 500 ppm
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:09.062297 17429 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:02:09.062753 17429 webserver.cc:460] Webserver started at http://127.0.98.1:39913/ using document root <none> and password file <none>
I20260929 19:02:09.063021 17429 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.063090 17429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.063328 17429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.064435 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "5214f250ee684fffa1004fc1a2a8ca3a"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0907b797070100aa43724562603e1966"
server_key_iv: "67dd5822873e452377890a03368c995b"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.064774 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5214f250ee684fffa1004fc1a2a8ca3a"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0907b797070100aa43724562603e1966"
server_key_iv: "67dd5822873e452377890a03368c995b"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.066198 17429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:09.067102 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.067385 17429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:09.067502 17429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "5214f250ee684fffa1004fc1a2a8ca3a"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0907b797070100aa43724562603e1966"
server_key_iv: "67dd5822873e452377890a03368c995b"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.067598 17429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:09.083241 17429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:09.085829 17429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:09.086005 17429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:09.086252 17429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:09.086575 17429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:09.086632 17429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.086679 17429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:09.086716 17429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.093299 17429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46799
I20260929 19:02:09.093359 17557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46799 every 8 connection(s)
I20260929 19:02:09.093747 17429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:09.095809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17429
I20260929 19:02:09.096035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:09.096287 392 external_mini_cluster.cc:1428] Setting key 232d9dbd2d2b2a8069586f484a14334c
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:02:09.105293 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.094608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49411 (local address 127.0.98.62:43427)
0929 19:02:09.094681 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.094683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.094930 (+ 247us) server_negotiation.cc:407] Connection header received
0929 19:02:09.096136 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.096138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.096156 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.096226 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.096901 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.097659 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.098808 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.099167 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.099256 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.101733 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.101735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.101735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:09.101745 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.103493 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.103997 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.103998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.103999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.104056 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.104482 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.104483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.104483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.104600 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.104661 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.104999 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.105226 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:02:09.105445 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.094587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.1:49411)
0929 19:02:09.094795 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.094831 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.096020 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.096320 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.096325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.096642 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:09.096830 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.096835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.097763 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.097764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.098712 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.098715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.099326 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.099328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.099449 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.100012 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:09.100020 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:09.101614 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:09.103574 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.103582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.103584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.103887 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.104244 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.104245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.104245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.104392 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.104684 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:09.104692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:09.104850 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.104980 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.105159 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260929 19:02:09.106029 17558 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260929 19:02:09.106112 17558 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:09.106299 17558 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260929 19:02:09.106825 17367 ts_manager.cc:194] Registered new tserver with Master: 5214f250ee684fffa1004fc1a2a8ca3a (127.0.98.1:46799)
I20260929 19:02:09.107690 17367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:02:09.123353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:09.206326 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.206566 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.206605 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.207885 17563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.207944 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.207996 17563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:09.209486 17563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.209746 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.209986 17563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.215077 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.215123 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.215095 17568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.215128 17563 server_base.cc:1034] running on GCE node
I20260929 19:02:09.215577 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.215806 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.216979 17563 hybrid_clock.cc:648] HybridClock initialized: now 1790708529216962 us; error 32 us; skew 500 ppm
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:09.219630 17563 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:02:09.220009 17563 webserver.cc:460] Webserver started at http://127.0.98.2:37617/ using document root <none> and password file <none>
I20260929 19:02:09.220183 17563 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.220222 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.220310 17563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.221278 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae04d3dea8664ae99e1fd9f882f96a91"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "88920d93f54cf0ed5defbb58b331e7e8"
server_key_iv: "959f2c94d9c56a7b6a0352349244a152"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.221592 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae04d3dea8664ae99e1fd9f882f96a91"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "88920d93f54cf0ed5defbb58b331e7e8"
server_key_iv: "959f2c94d9c56a7b6a0352349244a152"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.222918 17563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:09.223666 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.223850 17563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:02:09.223953 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "ae04d3dea8664ae99e1fd9f882f96a91"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "88920d93f54cf0ed5defbb58b331e7e8"
server_key_iv: "959f2c94d9c56a7b6a0352349244a152"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.224037 17563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:09.239108 17563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:09.241300 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:09.241470 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:09.241717 17563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:09.242031 17563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:09.242085 17563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.242129 17563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:09.242154 17563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.248112 17563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33755
I20260929 19:02:09.248157 17691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33755 every 8 connection(s)
I20260929 19:02:09.248486 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:09.257911 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.249225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50759 (local address 127.0.98.62:43427)
0929 19:02:09.249303 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.249305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.249454 (+ 149us) server_negotiation.cc:407] Connection header received
0929 19:02:09.250379 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.250381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.250391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.250481 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.251120 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.251740 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.252740 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.252932 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.252996 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.255247 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.255249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.255249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:09.255255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.256653 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.257007 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.257008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.257008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.257060 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.257307 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.257309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.257309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.257467 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.257529 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.257768 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.257843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:02:09.258051 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.249177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.2:50759)
0929 19:02:09.249336 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.249372 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.250286 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.250560 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.250564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.250841 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:09.251029 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.251032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.251820 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.251821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.252656 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.252658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.253008 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.253009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.253129 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.253698 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:09.253707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:09.255144 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:09.256726 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.256732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.256734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.256940 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.257120 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.257121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.257121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.257223 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.257529 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:09.257533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:09.257648 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.257730 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.257845 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260929 19:02:09.258545 17692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260929 19:02:09.258566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17563
I20260929 19:02:09.258679 17692 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:09.258689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:09.258872 17692 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260929 19:02:09.258919 392 external_mini_cluster.cc:1428] Setting key a2b827b9df66dac777c59172991bcdc2
I20260929 19:02:09.259341 17367 ts_manager.cc:194] Registered new tserver with Master: ae04d3dea8664ae99e1fd9f882f96a91 (127.0.98.2:33755)
I20260929 19:02:09.259838 17367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50759
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:02:09.285750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:09.376673 17697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.376972 17697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.377012 17697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.379050 17697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.379112 17697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.379158 17697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:09.381742 17697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.381999 17697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.382263 17697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.387871 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.387976 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.387924 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.388748 17697 server_base.cc:1034] running on GCE node
I20260929 19:02:09.389008 17697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.389257 17697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.390456 17697 hybrid_clock.cc:648] HybridClock initialized: now 1790708529390453 us; error 40 us; skew 500 ppm
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:09.393126 17697 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:02:09.393597 17697 webserver.cc:460] Webserver started at http://127.0.98.3:45375/ using document root <none> and password file <none>
I20260929 19:02:09.393839 17697 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.393908 17697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.394062 17697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.395170 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "0baa44bafb024a74938ca8a54b270ec3"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "9d90c93271e4a295a71a2b2d6f836959"
server_key_iv: "e6ae24754fb645441c62a42eac0fb22e"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.395620 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0baa44bafb024a74938ca8a54b270ec3"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "9d90c93271e4a295a71a2b2d6f836959"
server_key_iv: "e6ae24754fb645441c62a42eac0fb22e"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.397109 17697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:09.397912 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.398111 17697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:09.398211 17697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "0baa44bafb024a74938ca8a54b270ec3"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "9d90c93271e4a295a71a2b2d6f836959"
server_key_iv: "e6ae24754fb645441c62a42eac0fb22e"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.398303 17697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:09.421559 17697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:09.424090 17697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:09.424256 17697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:09.424525 17697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:09.424836 17697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:09.424888 17697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.424923 17697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:09.424968 17697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.430963 17697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39999
I20260929 19:02:09.431012 17825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39999 every 8 connection(s)
I20260929 19:02:09.431334 17697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:09.434484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17697
I20260929 19:02:09.434598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:09.434811 392 external_mini_cluster.cc:1428] Setting key b7bae3185bce88bf8d30010745a94373
Sep 29 19:02:09 dist-test-slave-gfsd krb5kdc[17322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:09.441388 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.432069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45539 (local address 127.0.98.62:43427)
0929 19:02:09.432171 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.432173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.432380 (+ 207us) server_negotiation.cc:407] Connection header received
0929 19:02:09.433336 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.433337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.433348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.433417 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.434027 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.434567 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.436099 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.436284 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.436341 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.438578 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.438580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.438580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:09.438586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.440022 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.440440 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.440441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.440441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.440492 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.440717 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.440718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.440718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.440813 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.440861 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.441247 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.441315 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 19:02:09.441567 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.432037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.3:45539)
0929 19:02:09.432207 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.432250 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.433240 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.433488 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.433491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.433752 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:09.433946 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.433951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.434698 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.434699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.436017 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.436020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.436354 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.436355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.436455 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.437022 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:09.437030 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:09.438470 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:09.440117 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.440123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.440125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.440369 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.440557 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:09.440557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:09.440558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:09.440659 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:09.440877 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:09.440883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:09.441101 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.441196 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.441321 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260929 19:02:09.442157 17826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260929 19:02:09.442242 17826 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:09.442373 17826 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260929 19:02:09.442798 17367 ts_manager.cc:194] Registered new tserver with Master: 0baa44bafb024a74938ca8a54b270ec3 (127.0.98.3:39999)
I20260929 19:02:09.443150 17367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45539
I20260929 19:02:09.446714 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:09.454133 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.448181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.62:43427)
0929 19:02:09.448243 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.448245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.448254 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:09.448274 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.448275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.448284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.448337 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.448898 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.449364 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.450095 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.450300 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.450371 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.451611 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.451612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.451613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:09.451619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.452967 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.453321 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.453322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.453322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.453381 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:09.453597 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:09.453598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:09.453599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:09.453746 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.453849 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.453954 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.454031 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:02:09.455744 17367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:09.457190 17367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:09.464157 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.0.98.3:39999)
0929 19:02:09.460155 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.460159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.460168 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:09.460214 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.460216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.460231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.460314 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:09.460424 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461485 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.462925 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463717 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463808 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.463868 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.464030 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.464071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":63,"threads_started":1}
I20260929 19:02:09.464165 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39999 (local address 127.0.0.1:51054)
0929 19:02:09.459724 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.459733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.459775 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.460310 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.460311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.460313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:09.460359 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.460361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461608 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.461609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.462805 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.462808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463811 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.463891 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.463904 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.464046 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.464087 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":59,"thread_start_us":96,"threads_started":1}
I20260929 19:02:09.464253 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.0.98.2:33755)
0929 19:02:09.459864 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.459868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.460496 (+ 628us) server_negotiation.cc:407] Connection header received
0929 19:02:09.460566 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.460568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.460582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.460674 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:09.460785 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461463 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.462931 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463728 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.463868 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.464129 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.464171 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260929 19:02:09.464371 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33755 (local address 127.0.0.1:54708)
0929 19:02:09.460434 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.460444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.460492 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.460682 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.460683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.460685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:09.460722 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.460731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461566 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.461567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.462824 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.462828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463896 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.463948 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.463951 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.463997 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.464027 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260929 19:02:09.464452 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.0.98.1:46799)
0929 19:02:09.459820 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.459826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.459835 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:09.459873 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.459875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.459901 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.460004 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:09.460165 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461314 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463054 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463734 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.464145 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.464215 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.464239 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.464284 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260929 19:02:09.464380 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.459373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46799 (local address 127.0.0.1:44214)
0929 19:02:09.459501 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.459541 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.459599 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.460017 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.460021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.460023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:09.460092 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.460094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.461429 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.461430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.462825 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.462828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.463849 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.463850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.463927 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.463931 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.463965 (+ 34us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.464027 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":76,"thread_start_us":84,"threads_started":1}
I20260929 19:02:09.465304 17492 tablet_service.cc:1467] Processing CreateTablet for tablet 61c449a4a80f40ffb84f92725a725684 (DEFAULT_TABLE table=test-table [id=4aa99037027848ffb53a042d076124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:09.465322 17626 tablet_service.cc:1467] Processing CreateTablet for tablet 61c449a4a80f40ffb84f92725a725684 (DEFAULT_TABLE table=test-table [id=4aa99037027848ffb53a042d076124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:09.465693 17492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c449a4a80f40ffb84f92725a725684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:09.465700 17626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c449a4a80f40ffb84f92725a725684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:09.468245 17843 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: Bootstrap starting.
I20260929 19:02:09.468567 17760 tablet_service.cc:1467] Processing CreateTablet for tablet 61c449a4a80f40ffb84f92725a725684 (DEFAULT_TABLE table=test-table [id=4aa99037027848ffb53a042d076124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:09.468830 17843 tablet_bootstrap.cc:654] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:09.468863 17760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c449a4a80f40ffb84f92725a725684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:09.469321 17843 log.cc:824] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:09.470037 17843 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: No bootstrap required, opened a new log
I20260929 19:02:09.470089 17843 ts_tablet_manager.cc:1397] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:09.471412 17846 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Bootstrap starting.
I20260929 19:02:09.471698 17844 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Bootstrap starting.
I20260929 19:02:09.471802 17843 raft_consensus.cc:348] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.472015 17843 raft_consensus.cc:374] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:09.472080 17843 raft_consensus.cc:729] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5214f250ee684fffa1004fc1a2a8ca3a, State: Initialized, Role: FOLLOWER
I20260929 19:02:09.472105 17846 tablet_bootstrap.cc:654] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:09.472218 17843 consensus_queue.cc:260] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.472375 17844 tablet_bootstrap.cc:654] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:09.472517 17558 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
I20260929 19:02:09.472515 17843 ts_tablet_manager.cc:1428] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:02:09.472716 17846 log.cc:824] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:09.473069 17844 log.cc:824] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:09.473771 17846 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: No bootstrap required, opened a new log
I20260929 19:02:09.473795 17844 tablet_bootstrap.cc:492] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: No bootstrap required, opened a new log
I20260929 19:02:09.473845 17844 ts_tablet_manager.cc:1397] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:09.473845 17846 ts_tablet_manager.cc:1397] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:09.475835 17846 raft_consensus.cc:348] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.475835 17844 raft_consensus.cc:348] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.476015 17844 raft_consensus.cc:374] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:09.476055 17846 raft_consensus.cc:374] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:09.476080 17844 raft_consensus.cc:729] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae04d3dea8664ae99e1fd9f882f96a91, State: Initialized, Role: FOLLOWER
I20260929 19:02:09.476109 17846 raft_consensus.cc:729] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0baa44bafb024a74938ca8a54b270ec3, State: Initialized, Role: FOLLOWER
I20260929 19:02:09.476209 17846 consensus_queue.cc:260] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.476234 17844 consensus_queue.cc:260] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.476504 17846 ts_tablet_manager.cc:1428] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:09.476501 17844 ts_tablet_manager.cc:1428] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:02:09.476517 17692 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
I20260929 19:02:09.476567 17826 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
W20260929 19:02:09.499023 17693 tablet.cc:2369] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:09.525590 17851 raft_consensus.cc:482] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:09.525727 17851 raft_consensus.cc:504] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.526222 17851 leader_election.cc:288] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5214f250ee684fffa1004fc1a2a8ca3a (127.0.98.1:46799), 0baa44bafb024a74938ca8a54b270ec3 (127.0.98.3:39999)
I20260929 19:02:09.530146 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.526474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54881 (local address 127.0.98.1:46799)
0929 19:02:09.526535 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.526537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.526547 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:09.526603 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.526604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.526616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.526675 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:09.526872 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.527745 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529169 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.529735 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529814 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.529824 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.530012 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.530077 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:02:09.530171 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.526639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48111 (local address 127.0.98.3:39999)
0929 19:02:09.526721 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.526724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.526854 (+ 130us) server_negotiation.cc:407] Connection header received
0929 19:02:09.526886 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.526887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.526896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.526972 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:09.527137 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.527878 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529170 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.529738 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529814 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.529824 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.530067 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.530108 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:02:09.530180 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.526430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46799 (local address 127.0.98.2:54881)
0929 19:02:09.526488 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.526501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.526547 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.526683 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.526684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.526685 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:09.526758 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.526760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.527832 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.527833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.529062 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.529065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.529833 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.529928 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.529932 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.529995 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.530044 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":9}
I20260929 19:02:09.530186 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.526563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39999 (local address 127.0.98.2:48111)
0929 19:02:09.526773 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.526783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.526838 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.526980 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.526980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.526981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:09.527060 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.527061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.527960 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.527960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.529063 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.529065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.529833 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.529834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.529989 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.529992 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.530044 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.530071 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":5,"thread_start_us":127,"threads_started":1}
I20260929 19:02:09.530536 17512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c449a4a80f40ffb84f92725a725684" candidate_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" is_pre_election: true
I20260929 19:02:09.530704 17512 raft_consensus.cc:2454] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae04d3dea8664ae99e1fd9f882f96a91 in term 0.
I20260929 19:02:09.530839 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c449a4a80f40ffb84f92725a725684" candidate_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0baa44bafb024a74938ca8a54b270ec3" is_pre_election: true
I20260929 19:02:09.530932 17579 leader_election.cc:302] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5214f250ee684fffa1004fc1a2a8ca3a, ae04d3dea8664ae99e1fd9f882f96a91; no voters:
I20260929 19:02:09.531019 17780 raft_consensus.cc:2454] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae04d3dea8664ae99e1fd9f882f96a91 in term 0.
I20260929 19:02:09.531060 17851 raft_consensus.cc:2781] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:09.531116 17851 raft_consensus.cc:482] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:09.531157 17851 raft_consensus.cc:3037] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:09.531869 17851 raft_consensus.cc:504] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.532033 17851 leader_election.cc:288] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [CANDIDATE]: Term 1 election: Requested vote from peers 5214f250ee684fffa1004fc1a2a8ca3a (127.0.98.1:46799), 0baa44bafb024a74938ca8a54b270ec3 (127.0.98.3:39999)
I20260929 19:02:09.532212 17512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c449a4a80f40ffb84f92725a725684" candidate_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5214f250ee684fffa1004fc1a2a8ca3a"
I20260929 19:02:09.532266 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c449a4a80f40ffb84f92725a725684" candidate_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0baa44bafb024a74938ca8a54b270ec3"
I20260929 19:02:09.532300 17512 raft_consensus.cc:3037] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:09.532344 17780 raft_consensus.cc:3037] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:09.533141 17512 raft_consensus.cc:2454] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae04d3dea8664ae99e1fd9f882f96a91 in term 1.
I20260929 19:02:09.533167 17780 raft_consensus.cc:2454] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae04d3dea8664ae99e1fd9f882f96a91 in term 1.
I20260929 19:02:09.533319 17579 leader_election.cc:302] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5214f250ee684fffa1004fc1a2a8ca3a, ae04d3dea8664ae99e1fd9f882f96a91; no voters:
I20260929 19:02:09.533409 17851 raft_consensus.cc:2781] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:09.533617 17851 raft_consensus.cc:686] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 1 LEADER]: Becoming Leader. State: Replica: ae04d3dea8664ae99e1fd9f882f96a91, State: Running, Role: LEADER
I20260929 19:02:09.533747 17851 consensus_queue.cc:237] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } }
I20260929 19:02:09.534679 17366 catalog_manager.cc:5495] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae04d3dea8664ae99e1fd9f882f96a91 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d3dea8664ae99e1fd9f882f96a91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:09.555855 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.553443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.0.98.2:33755)
0929 19:02:09.553508 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.553514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.553523 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:09.553558 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.553559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.553577 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.553641 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:09.553804 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.554318 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.555058 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.555237 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.555313 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.555503 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:09.555597 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:09.555661 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.555699 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.555718 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:02:09.557535 17512 raft_consensus.cc:1261] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 1 FOLLOWER]: Refusing update from remote peer ae04d3dea8664ae99e1fd9f882f96a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:09.557571 17780 raft_consensus.cc:1261] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 1 FOLLOWER]: Refusing update from remote peer ae04d3dea8664ae99e1fd9f882f96a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:09.557917 17851 consensus_queue.cc:1035] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5214f250ee684fffa1004fc1a2a8ca3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:09.558051 17853 consensus_queue.cc:1035] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0baa44bafb024a74938ca8a54b270ec3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:09.560335 17856 mvcc.cc:205] Tried to move back new op lower bound from 7334742137065287680 to 7334742136971190272. Current Snapshot: MvccSnapshot[applied={T|T < 7334742137065287680}]
I20260929 19:02:09.560874 17858 mvcc.cc:205] Tried to move back new op lower bound from 7334742137065287680 to 7334742136971190272. Current Snapshot: MvccSnapshot[applied={T|T < 7334742137065287680}]
I20260929 19:02:09.563045 17365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:09.563155 17365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:09.564086 17365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0d5b7b1b86e48f5bbba5e92e25d6d54: Sending DeleteTablet for 3 replicas of tablet 61c449a4a80f40ffb84f92725a725684
I20260929 19:02:09.564464 17492 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c449a4a80f40ffb84f92725a725684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44214
I20260929 19:02:09.564690 17865 tablet_replica.cc:330] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a: stopping tablet replica
I20260929 19:02:09.564764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17429
I20260929 19:02:09.564785 17865 raft_consensus.cc:2229] T 61c449a4a80f40ffb84f92725a725684 P 5214f250ee684fffa1004fc1a2a8ca3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:09.564760 17760 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c449a4a80f40ffb84f92725a725684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51054
I20260929 19:02:09.567102 17626 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c449a4a80f40ffb84f92725a725684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54708
I20260929 19:02:09.567423 17866 tablet_replica.cc:330] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: stopping tablet replica
I20260929 19:02:09.567528 17866 raft_consensus.cc:2229] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:09.567623 17866 raft_consensus.cc:2258] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:09.567777 17867 tablet_replica.cc:330] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: stopping tablet replica
I20260929 19:02:09.567857 17867 raft_consensus.cc:2229] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:09.567972 17867 raft_consensus.cc:2258] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:09.568130 17866 ts_tablet_manager.cc:1905] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:09.568420 17867 ts_tablet_manager.cc:1905] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:09.570158 17866 ts_tablet_manager.cc:1918] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:09.570175 17867 ts_tablet_manager.cc:1918] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:09.570225 17866 log.cc:1196] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/61c449a4a80f40ffb84f92725a725684
I20260929 19:02:09.570238 17867 log.cc:1196] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/61c449a4a80f40ffb84f92725a725684
I20260929 19:02:09.570515 17867 ts_tablet_manager.cc:1939] T 61c449a4a80f40ffb84f92725a725684 P ae04d3dea8664ae99e1fd9f882f96a91: Deleting consensus metadata
I20260929 19:02:09.570518 17866 ts_tablet_manager.cc:1939] T 61c449a4a80f40ffb84f92725a725684 P 0baa44bafb024a74938ca8a54b270ec3: Deleting consensus metadata
I20260929 19:02:09.570910 17351 catalog_manager.cc:4841] TS ae04d3dea8664ae99e1fd9f882f96a91 (127.0.98.2:33755): tablet 61c449a4a80f40ffb84f92725a725684 (table test-table [id=4aa99037027848ffb53a042d076124a8]) successfully deleted
I20260929 19:02:09.570923 17354 catalog_manager.cc:4841] TS 0baa44bafb024a74938ca8a54b270ec3 (127.0.98.3:39999): tablet 61c449a4a80f40ffb84f92725a725684 (table test-table [id=4aa99037027848ffb53a042d076124a8]) successfully deleted
W20260929 19:02:09.571085 17351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46799 (error 108)
I20260929 19:02:09.571566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17563
I20260929 19:02:09.571592 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.571465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46799 (local address 127.0.0.1:44216)
0929 19:02:09.571519 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.571555 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:02:09.571719 17351 catalog_manager.cc:4584] TS 5214f250ee684fffa1004fc1a2a8ca3a (127.0.98.1:46799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61c449a4a80f40ffb84f92725a725684: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46799: connect: Connection refused (error 111)
I20260929 19:02:09.576411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17697
I20260929 19:02:09.581185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17335
2026-09-29T19:02:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T19:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:09Z Disabled control of system clock
I20260929 19:02:09.607579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35135
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:09.685658 17872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.685864 17872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.685885 17872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.686949 17872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:09.686981 17872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.686995 17872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.687007 17872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:09.687021 17872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:09.688306 17872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.688485 17872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.688695 17872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.693576 17878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.693624 17872 server_base.cc:1034] running on GCE node
W20260929 19:02:09.693593 17880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.693593 17877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.694103 17872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.694397 17872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.695587 17872 hybrid_clock.cc:648] HybridClock initialized: now 1790708529695565 us; error 47 us; skew 500 ppm
I20260929 19:02:09.696941 17872 webserver.cc:460] Webserver started at http://127.0.98.62:45301/ using document root <none> and password file <none>
I20260929 19:02:09.697283 17872 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.697383 17872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.697543 17872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.698650 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "12b6b27c4e8548d6a7417e639d2dbc6b"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "eaf97b8c455bb6c81a7cf8c0ed45f481"
server_key_iv: "e432405d6f88f4366c370bd990b53edd"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.699028 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "12b6b27c4e8548d6a7417e639d2dbc6b"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "eaf97b8c455bb6c81a7cf8c0ed45f481"
server_key_iv: "e432405d6f88f4366c370bd990b53edd"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.700467 17872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:09.701251 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.701442 17872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:09.701548 17872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "12b6b27c4e8548d6a7417e639d2dbc6b"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "eaf97b8c455bb6c81a7cf8c0ed45f481"
server_key_iv: "e432405d6f88f4366c370bd990b53edd"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.701637 17872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:09.718838 17872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:09.719154 17872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:09.719395 17872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:09.723448 17872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35135
I20260929 19:02:09.723481 17938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35135 every 8 connection(s)
I20260929 19:02:09.723831 17872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:09.724759 17939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:09.726938 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: Bootstrap starting.
I20260929 19:02:09.727597 17939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:09.727834 17939 log.cc:824] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:09.728453 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: No bootstrap required, opened a new log
I20260929 19:02:09.729924 17939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260929 19:02:09.730068 17939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:09.730089 17939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b6b27c4e8548d6a7417e639d2dbc6b, State: Initialized, Role: FOLLOWER
I20260929 19:02:09.730262 17939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260929 19:02:09.730343 17939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:09.730365 17939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:09.730392 17939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:09.730846 17939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260929 19:02:09.730981 17939 leader_election.cc:302] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b6b27c4e8548d6a7417e639d2dbc6b; no voters:
I20260929 19:02:09.731158 17939 leader_election.cc:288] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:09.731207 17942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:09.731348 17942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [term 1 LEADER]: Becoming Leader. State: Replica: 12b6b27c4e8548d6a7417e639d2dbc6b, State: Running, Role: LEADER
I20260929 19:02:09.731472 17939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:09.731448 17942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260929 19:02:09.731696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17872
I20260929 19:02:09.731822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:09.731803 17944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b6b27c4e8548d6a7417e639d2dbc6b. Latest consensus state: current_term: 1 leader_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } } }
I20260929 19:02:09.731871 17944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:09.732046 392 external_mini_cluster.cc:1428] Setting key c0d351a66f719ce23056d2eac76fdeab
I20260929 19:02:09.732085 17946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:09.732810 17946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:09.733560 17943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b6b27c4e8548d6a7417e639d2dbc6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } } }
I20260929 19:02:09.733657 17943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:09.734622 17946 catalog_manager.cc:1347] Generated new cluster ID: af6a9b334b824a17930b47a544086826
I20260929 19:02:09.734684 17946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:09.738464 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.732804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.0.98.62:35135)
0929 19:02:09.732990 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.732998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.733028 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:02:09.733140 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.733147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.733216 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.733365 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.733763 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.735099 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.736192 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.736823 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.736952 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.737656 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.737661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.737662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:09.737866 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.737908 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:09.737922 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.737994 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.738059 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.738242 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260929 19:02:09.763019 17946 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:09.763787 17946 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:09.768918 17946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: Generated new TSK 0
I20260929 19:02:09.769110 17946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:09.777079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--builtin_ntp_servers=127.0.98.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:09.859558 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.859795 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.859830 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.861013 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.861078 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.861117 17963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:09.862481 17963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.862751 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.863058 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.868096 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.868124 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.868175 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.868432 17963 server_base.cc:1034] running on GCE node
I20260929 19:02:09.868629 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.868878 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.870066 17963 hybrid_clock.cc:648] HybridClock initialized: now 1790708529870055 us; error 29 us; skew 500 ppm
I20260929 19:02:09.871501 17963 webserver.cc:460] Webserver started at http://127.0.98.1:41199/ using document root <none> and password file <none>
I20260929 19:02:09.871743 17963 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.871814 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.871994 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.873304 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "91a4bccd9dc44a59a3536f264fca0cc1"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "f877fd4ce279cecd6070519dda7ec7b4"
server_key_iv: "d9497dab7e2f31d4efb1f8baa5049501"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.873706 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91a4bccd9dc44a59a3536f264fca0cc1"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "f877fd4ce279cecd6070519dda7ec7b4"
server_key_iv: "d9497dab7e2f31d4efb1f8baa5049501"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.875123 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:09.876015 17977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.876194 17963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:09.876293 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "91a4bccd9dc44a59a3536f264fca0cc1"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "f877fd4ce279cecd6070519dda7ec7b4"
server_key_iv: "d9497dab7e2f31d4efb1f8baa5049501"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.876375 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:09.891765 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:09.892074 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:09.892241 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:09.892478 17963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:09.892809 17963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:09.892865 17963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.892905 17963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:09.892954 17963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:09.899151 17963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39335
I20260929 19:02:09.899199 18090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39335 every 8 connection(s)
I20260929 19:02:09.899672 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:09.902241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 17963
I20260929 19:02:09.902362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:09.902601 392 external_mini_cluster.cc:1428] Setting key d25dd766c853e4e74a5a7bb7f054ed9e
I20260929 19:02:09.903718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--builtin_ntp_servers=127.0.98.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:02:09.906453 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.900462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56509 (local address 127.0.98.62:35135)
0929 19:02:09.900510 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:09.900511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:09.900801 (+ 290us) server_negotiation.cc:407] Connection header received
0929 19:02:09.901448 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:09.901450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:09.901474 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:09.901533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:09.902456 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.903269 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.904915 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.905127 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.905215 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.906150 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:09.906152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:09.906152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:09.906157 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:09.906172 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:09.906176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:09.906228 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:09.906342 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 19:02:09.906375 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:09.906692 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:09.900480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.1:56509)
0929 19:02:09.900680 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:09.900713 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:09.901336 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:09.901613 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:09.901616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:09.901625 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:09.902275 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.902282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.903437 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.903438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.904832 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:09.904835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:09.905224 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:09.905226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:09.905366 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:09.906002 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:09.906002 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:09.906035 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:09.906059 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:09.906239 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:09.906245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:09.906248 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:09.906330 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:02:09.906435 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260929 19:02:09.907215 18091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260929 19:02:09.907377 18091 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:09.907665 18091 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260929 19:02:09.908439 17903 ts_manager.cc:194] Registered new tserver with Master: 91a4bccd9dc44a59a3536f264fca0cc1 (127.0.98.1:39335)
I20260929 19:02:09.909078 17903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56509
W20260929 19:02:09.985594 18094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:09.985792 18094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:09.985814 18094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:09.987056 18094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:09.987098 18094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:09.987124 18094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:09.988474 18094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:09.988701 18094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:09.988993 18094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:09.994095 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:09.994092 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.994119 18094 server_base.cc:1034] running on GCE node
W20260929 19:02:09.994093 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:09.994553 18094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:09.994746 18094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:09.995880 18094 hybrid_clock.cc:648] HybridClock initialized: now 1790708529995865 us; error 34 us; skew 500 ppm
I20260929 19:02:09.997080 18094 webserver.cc:460] Webserver started at http://127.0.98.2:43075/ using document root <none> and password file <none>
I20260929 19:02:09.997316 18094 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:09.997359 18094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:09.997538 18094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:09.998591 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "83c7f7a18f994237bd59f4cb2ada9624"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0f801abbcb791d1e37a6692a9440162c"
server_key_iv: "dac8e9a1712b2e61e1232e018756a19a"
server_key_version: "encryptionkey@0"
I20260929 19:02:09.998960 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83c7f7a18f994237bd59f4cb2ada9624"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0f801abbcb791d1e37a6692a9440162c"
server_key_iv: "dac8e9a1712b2e61e1232e018756a19a"
server_key_version: "encryptionkey@0"
I20260929 19:02:10.000797 18094 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 19:02:10.001618 18108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.001969 18094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:10.002064 18094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "83c7f7a18f994237bd59f4cb2ada9624"
format_stamp: "Formatted at 2026-09-29 19:02:09 on dist-test-slave-gfsd"
server_key: "0f801abbcb791d1e37a6692a9440162c"
server_key_iv: "dac8e9a1712b2e61e1232e018756a19a"
server_key_version: "encryptionkey@0"
I20260929 19:02:10.002136 18094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:10.012637 18094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:10.012923 18094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:10.013055 18094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:10.013285 18094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:10.013607 18094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:10.013662 18094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.013705 18094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:10.013732 18094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.019271 18094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38229
I20260929 19:02:10.019342 18221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38229 every 8 connection(s)
I20260929 19:02:10.019670 18094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:10.024932 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.020361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33653 (local address 127.0.98.62:35135)
0929 19:02:10.020427 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.020430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.020665 (+ 235us) server_negotiation.cc:407] Connection header received
0929 19:02:10.021186 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.021187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.021200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.021261 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:10.021960 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.022461 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.023536 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.023741 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.023827 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.024639 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:10.024641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:10.024641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:10.024646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:10.024661 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:10.024666 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:10.024719 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.024816 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.024846 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:02:10.025130 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.020334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.2:33653)
0929 19:02:10.020547 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.020580 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.021079 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.021340 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.021344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.021353 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:10.021841 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.021845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.022558 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.022559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.023438 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.023441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.023849 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.023850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.023989 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.024504 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:10.024504 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:10.024531 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:10.024552 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:10.024726 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:10.024730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:10.024733 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.024818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.024915 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260929 19:02:10.025645 18222 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260929 19:02:10.025739 18222 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:10.025879 18222 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260929 19:02:10.026248 17903 ts_manager.cc:194] Registered new tserver with Master: 83c7f7a18f994237bd59f4cb2ada9624 (127.0.98.2:38229)
I20260929 19:02:10.026625 17903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33653
I20260929 19:02:10.029292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18094
I20260929 19:02:10.029407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:10.029611 392 external_mini_cluster.cc:1428] Setting key 25aa3091e15337341d8c4300be6a3c06
I20260929 19:02:10.030566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--builtin_ntp_servers=127.0.98.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:10.113523 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:10.113730 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:10.113751 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:10.114897 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:10.114938 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:10.114964 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:10.116431 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:10.116683 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:10.116988 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:10.122120 18231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:10.122159 18233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:10.122180 18225 server_base.cc:1034] running on GCE node
W20260929 19:02:10.122135 18230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:10.122591 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:10.122833 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:10.123997 18225 hybrid_clock.cc:648] HybridClock initialized: now 1790708530123982 us; error 38 us; skew 500 ppm
I20260929 19:02:10.125355 18225 webserver.cc:460] Webserver started at http://127.0.98.3:45779/ using document root <none> and password file <none>
I20260929 19:02:10.125597 18225 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:10.125665 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:10.125818 18225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:10.126868 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0e460412b8444fb97b8748018f5a6bc"
format_stamp: "Formatted at 2026-09-29 19:02:10 on dist-test-slave-gfsd"
server_key: "5a957e8ad00e4e374a7ac6eeccb63392"
server_key_iv: "bb149e5d837183bba038ecc035b503f8"
server_key_version: "encryptionkey@0"
I20260929 19:02:10.127226 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0e460412b8444fb97b8748018f5a6bc"
format_stamp: "Formatted at 2026-09-29 19:02:10 on dist-test-slave-gfsd"
server_key: "5a957e8ad00e4e374a7ac6eeccb63392"
server_key_iv: "bb149e5d837183bba038ecc035b503f8"
server_key_version: "encryptionkey@0"
I20260929 19:02:10.128520 18225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:10.129215 18239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.129384 18225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:10.129503 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "b0e460412b8444fb97b8748018f5a6bc"
format_stamp: "Formatted at 2026-09-29 19:02:10 on dist-test-slave-gfsd"
server_key: "5a957e8ad00e4e374a7ac6eeccb63392"
server_key_iv: "bb149e5d837183bba038ecc035b503f8"
server_key_version: "encryptionkey@0"
I20260929 19:02:10.129575 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:10.140368 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:10.140655 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:10.140801 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:10.141021 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:10.141335 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:10.141389 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.141429 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:10.141484 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:10.147343 18225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37883
I20260929 19:02:10.147373 18352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37883 every 8 connection(s)
I20260929 19:02:10.147732 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:10.153156 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.148541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32813 (local address 127.0.98.62:35135)
0929 19:02:10.148612 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.148614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.148859 (+ 245us) server_negotiation.cc:407] Connection header received
0929 19:02:10.149370 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.149371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.149380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.149437 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:10.150105 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.150659 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.151726 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.151936 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.152003 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.152814 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:10.152815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:10.152815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:10.152820 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:10.152834 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:10.152838 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:10.152898 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.153071 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.153097 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:02:10.153254 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.148519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.3:32813)
0929 19:02:10.148726 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.148768 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.149280 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.149501 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.149505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.149513 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:10.149997 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.150001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.150754 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.150755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.151646 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.151649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.152046 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.152047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.152163 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.152680 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:10.152680 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:10.152708 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:10.152732 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:10.152905 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:10.152909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:10.152912 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.152980 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.153061 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260929 19:02:10.153779 18353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260929 19:02:10.153869 18353 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:10.154067 18353 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260929 19:02:10.154475 17903 ts_manager.cc:194] Registered new tserver with Master: b0e460412b8444fb97b8748018f5a6bc (127.0.98.3:37883)
I20260929 19:02:10.154841 17903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:32813
I20260929 19:02:10.155589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18225
I20260929 19:02:10.155714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:10.155982 392 external_mini_cluster.cc:1428] Setting key 70bf54a0fa24641d6050ecc4e69c19b8
I20260929 19:02:10.157099 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:10.161268 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.158500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.0.98.62:35135)
0929 19:02:10.158560 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.158562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.158674 (+ 112us) server_negotiation.cc:407] Connection header received
0929 19:02:10.158729 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.158730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.158740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.158780 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:10.158942 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.159470 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.160222 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.160393 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.160440 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.161052 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:10.161053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:10.161053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:10.161057 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:10.161070 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:10.161074 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:10.161118 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.161190 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.161211 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:02:10.162853 17903 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:10.164484 17903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:10.170681 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.166760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37883 (local address 127.0.0.1:56364)
0929 19:02:10.166896 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.166943 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.167015 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.167672 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.167675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.167678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:10.167760 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.167763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.168665 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.168666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.169638 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.169640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.170346 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.170347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.170434 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.170447 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.170527 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.170596 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260929 19:02:10.170697 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.167063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.0.98.3:37883)
0929 19:02:10.167360 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.167365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.167375 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:10.167463 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.167464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.167478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.167581 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:10.167843 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.168571 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.169733 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.170251 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.170348 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.170461 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.170543 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.170596 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260929 19:02:10.171629 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.166789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38229 (local address 127.0.0.1:33388)
0929 19:02:10.167383 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.167395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.167480 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.167723 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.167724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.167726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:10.167781 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.167783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.169124 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.169125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.170309 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.170312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.171387 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.171388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.171453 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.171456 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.171513 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.171559 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":5,"thread_start_us":546,"threads_started":1}
I20260929 19:02:10.171679 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.166975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.0.98.1:39335)
0929 19:02:10.167297 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.167303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.167617 (+ 314us) server_negotiation.cc:407] Connection header received
0929 19:02:10.167807 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.167810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.167826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.167881 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:10.168074 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.169126 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.170419 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.171181 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.171231 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.171349 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.171550 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.171595 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":95,"threads_started":1}
I20260929 19:02:10.171692 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.167058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.98.2:38229)
0929 19:02:10.167252 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.167258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.167491 (+ 233us) server_negotiation.cc:407] Connection header received
0929 19:02:10.167554 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.167556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.167576 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.167651 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:10.167874 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.169025 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.170414 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.171181 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.171251 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.171349 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.171554 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.171604 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260929 19:02:10.171694 18287 tablet_service.cc:1467] Processing CreateTablet for tablet a08432073e084627ba1ebd6f5a74cb48 (DEFAULT_TABLE table=test-table [id=e5f36ff5a22f4e6fbd1f0af5ce0a6f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:10.172128 18287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08432073e084627ba1ebd6f5a74cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:10.171949 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.166824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39335 (local address 127.0.0.1:40106)
0929 19:02:10.167559 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.167568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.167709 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.167966 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.167966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.167968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:10.167998 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.167999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.169213 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.169214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.170312 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.170313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.171407 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.171408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.171457 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.171460 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.171841 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.171893 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":34,"thread_start_us":507,"threads_started":1}
I20260929 19:02:10.172797 18025 tablet_service.cc:1467] Processing CreateTablet for tablet a08432073e084627ba1ebd6f5a74cb48 (DEFAULT_TABLE table=test-table [id=e5f36ff5a22f4e6fbd1f0af5ce0a6f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:10.172883 18156 tablet_service.cc:1467] Processing CreateTablet for tablet a08432073e084627ba1ebd6f5a74cb48 (DEFAULT_TABLE table=test-table [id=e5f36ff5a22f4e6fbd1f0af5ce0a6f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:10.173162 18025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08432073e084627ba1ebd6f5a74cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:10.173241 18156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08432073e084627ba1ebd6f5a74cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:10.175900 18367 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: Bootstrap starting.
I20260929 19:02:10.176420 18369 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Bootstrap starting.
I20260929 19:02:10.176501 18367 tablet_bootstrap.cc:654] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:10.176795 18367 log.cc:824] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:10.176999 18369 tablet_bootstrap.cc:654] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:10.177137 18368 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Bootstrap starting.
I20260929 19:02:10.177320 18369 log.cc:824] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:10.177785 18368 tablet_bootstrap.cc:654] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:10.178073 18369 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: No bootstrap required, opened a new log
I20260929 19:02:10.178076 18368 log.cc:824] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:10.178153 18369 ts_tablet_manager.cc:1397] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:10.178617 18367 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: No bootstrap required, opened a new log
I20260929 19:02:10.178697 18367 ts_tablet_manager.cc:1397] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:02:10.178889 18368 tablet_bootstrap.cc:492] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: No bootstrap required, opened a new log
I20260929 19:02:10.179013 18368 ts_tablet_manager.cc:1397] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:10.180989 18369 raft_consensus.cc:348] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.181157 18369 raft_consensus.cc:374] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:10.181059 18367 raft_consensus.cc:348] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.181195 18369 raft_consensus.cc:729] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c7f7a18f994237bd59f4cb2ada9624, State: Initialized, Role: FOLLOWER
I20260929 19:02:10.181203 18367 raft_consensus.cc:374] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:10.181238 18367 raft_consensus.cc:729] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a4bccd9dc44a59a3536f264fca0cc1, State: Initialized, Role: FOLLOWER
I20260929 19:02:10.181367 18369 consensus_queue.cc:260] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.181394 18367 consensus_queue.cc:260] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.181628 18369 ts_tablet_manager.cc:1428] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:02:10.181672 18367 ts_tablet_manager.cc:1428] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:10.181628 18368 raft_consensus.cc:348] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.181790 18222 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
I20260929 19:02:10.181823 18368 raft_consensus.cc:374] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:10.181841 18091 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
I20260929 19:02:10.181860 18368 raft_consensus.cc:729] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e460412b8444fb97b8748018f5a6bc, State: Initialized, Role: FOLLOWER
I20260929 19:02:10.181991 18368 consensus_queue.cc:260] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.182299 18368 ts_tablet_manager.cc:1428] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:10.182530 18353 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
W20260929 19:02:10.270212 18223 tablet.cc:2369] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:10.299075 18374 raft_consensus.cc:482] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:10.299211 18374 raft_consensus.cc:504] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.299685 18374 leader_election.cc:288] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c7f7a18f994237bd59f4cb2ada9624 (127.0.98.2:38229), b0e460412b8444fb97b8748018f5a6bc (127.0.98.3:37883)
I20260929 19:02:10.303824 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.299866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38229 (local address 127.0.98.1:46791)
0929 19:02:10.299956 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.299969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.300018 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.300217 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.300218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.300220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:10.300318 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.300320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.301308 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.301309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.302678 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.302680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.303511 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.303512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.303596 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.303603 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.303666 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.303730 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":103}
I20260929 19:02:10.303838 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.299924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46791 (local address 127.0.98.2:38229)
0929 19:02:10.299984 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.299986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.300001 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:02:10.300101 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.300102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.300119 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.300195 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:10.300428 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.301165 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.302791 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.303419 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.303500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.303509 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.303722 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.303769 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:02:10.303838 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.299989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36579 (local address 127.0.98.3:37883)
0929 19:02:10.300048 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.300050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.300196 (+ 146us) server_negotiation.cc:407] Connection header received
0929 19:02:10.300230 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.300231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.300247 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.300298 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:10.300474 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.301215 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.302798 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.303422 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.303500 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.303509 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.303721 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.303769 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:02:10.303831 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.299921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37883 (local address 127.0.98.1:36579)
0929 19:02:10.300088 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.300098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:10.300157 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:10.300320 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:10.300320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:10.300322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:10.300397 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.300399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.301308 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.301309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.302674 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:10.302677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.303511 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.303512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:10.303596 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.303603 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:02:10.303666 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:02:10.303728 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":58,"thread_start_us":114,"threads_started":1}
I20260929 19:02:10.304241 18307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08432073e084627ba1ebd6f5a74cb48" candidate_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e460412b8444fb97b8748018f5a6bc" is_pre_election: true
I20260929 19:02:10.304242 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08432073e084627ba1ebd6f5a74cb48" candidate_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c7f7a18f994237bd59f4cb2ada9624" is_pre_election: true
I20260929 19:02:10.304391 18307 raft_consensus.cc:2454] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a4bccd9dc44a59a3536f264fca0cc1 in term 0.
I20260929 19:02:10.304422 18176 raft_consensus.cc:2454] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a4bccd9dc44a59a3536f264fca0cc1 in term 0.
I20260929 19:02:10.304641 17979 leader_election.cc:302] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a4bccd9dc44a59a3536f264fca0cc1, b0e460412b8444fb97b8748018f5a6bc; no voters:
I20260929 19:02:10.304824 18374 raft_consensus.cc:2781] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:10.304942 18374 raft_consensus.cc:482] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:10.304982 18374 raft_consensus.cc:3037] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:10.305820 18374 raft_consensus.cc:504] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.305967 18374 leader_election.cc:288] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [CANDIDATE]: Term 1 election: Requested vote from peers 83c7f7a18f994237bd59f4cb2ada9624 (127.0.98.2:38229), b0e460412b8444fb97b8748018f5a6bc (127.0.98.3:37883)
I20260929 19:02:10.306118 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08432073e084627ba1ebd6f5a74cb48" candidate_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c7f7a18f994237bd59f4cb2ada9624"
I20260929 19:02:10.306124 18307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08432073e084627ba1ebd6f5a74cb48" candidate_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e460412b8444fb97b8748018f5a6bc"
I20260929 19:02:10.306200 18176 raft_consensus.cc:3037] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:10.306221 18307 raft_consensus.cc:3037] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:10.306990 18307 raft_consensus.cc:2454] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a4bccd9dc44a59a3536f264fca0cc1 in term 1.
I20260929 19:02:10.307004 18176 raft_consensus.cc:2454] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a4bccd9dc44a59a3536f264fca0cc1 in term 1.
I20260929 19:02:10.307152 17979 leader_election.cc:302] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a4bccd9dc44a59a3536f264fca0cc1, b0e460412b8444fb97b8748018f5a6bc; no voters:
I20260929 19:02:10.307255 18374 raft_consensus.cc:2781] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:10.307448 18374 raft_consensus.cc:686] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 91a4bccd9dc44a59a3536f264fca0cc1, State: Running, Role: LEADER
I20260929 19:02:10.307560 18374 consensus_queue.cc:237] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } }
I20260929 19:02:10.308425 17903 catalog_manager.cc:5495] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91a4bccd9dc44a59a3536f264fca0cc1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91a4bccd9dc44a59a3536f264fca0cc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39335 } health_report { overall_health: HEALTHY } } }
I20260929 19:02:10.318758 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.316590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.98.1:39335)
0929 19:02:10.316640 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:10.316642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:10.316651 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:10.316681 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:10.316682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:10.316707 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:10.316769 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:10.316890 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.317334 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.318089 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:10.318259 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:10.318306 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:10.318487 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:10.318575 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:10.318633 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:10.318678 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:02:10.318694 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:02:10.320453 18176 raft_consensus.cc:1261] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 1 FOLLOWER]: Refusing update from remote peer 91a4bccd9dc44a59a3536f264fca0cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:10.320447 18307 raft_consensus.cc:1261] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 1 FOLLOWER]: Refusing update from remote peer 91a4bccd9dc44a59a3536f264fca0cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:10.320799 18374 consensus_queue.cc:1035] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c7f7a18f994237bd59f4cb2ada9624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:10.320956 18377 consensus_queue.cc:1035] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e460412b8444fb97b8748018f5a6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:10.323354 18381 mvcc.cc:205] Tried to move back new op lower bound from 7334742140190089216 to 7334742140140318720. Current Snapshot: MvccSnapshot[applied={T|T < 7334742140190089216}]
I20260929 19:02:10.323391 18382 mvcc.cc:205] Tried to move back new op lower bound from 7334742140190089216 to 7334742140140318720. Current Snapshot: MvccSnapshot[applied={T|T < 7334742140190089216}]
I20260929 19:02:10.323676 18380 mvcc.cc:205] Tried to move back new op lower bound from 7334742140190089216 to 7334742140140318720. Current Snapshot: MvccSnapshot[applied={T|T < 7334742140190089216}]
I20260929 19:02:10.325605 17903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:10.325687 17903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:10.326383 17903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12b6b27c4e8548d6a7417e639d2dbc6b: Sending DeleteTablet for 3 replicas of tablet a08432073e084627ba1ebd6f5a74cb48
I20260929 19:02:10.326689 18287 tablet_service.cc:1514] Processing DeleteTablet for tablet a08432073e084627ba1ebd6f5a74cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:10 UTC) from {username='slave'} at 127.0.0.1:56364
I20260929 19:02:10.326709 18156 tablet_service.cc:1514] Processing DeleteTablet for tablet a08432073e084627ba1ebd6f5a74cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:10 UTC) from {username='slave'} at 127.0.0.1:33388
I20260929 19:02:10.326894 18025 tablet_service.cc:1514] Processing DeleteTablet for tablet a08432073e084627ba1ebd6f5a74cb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:10 UTC) from {username='slave'} at 127.0.0.1:40106
I20260929 19:02:10.327102 18392 tablet_replica.cc:330] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1: stopping tablet replica
I20260929 19:02:10.327205 18392 raft_consensus.cc:2229] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:10.327332 18391 tablet_replica.cc:330] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: stopping tablet replica
I20260929 19:02:10.327350 18392 raft_consensus.cc:2258] T a08432073e084627ba1ebd6f5a74cb48 P 91a4bccd9dc44a59a3536f264fca0cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:10.327442 18391 raft_consensus.cc:2229] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:10.327436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17963
I20260929 19:02:10.331435 18391 raft_consensus.cc:2258] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:10.331672 18390 tablet_replica.cc:330] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: stopping tablet replica
I20260929 19:02:10.331792 18390 raft_consensus.cc:2229] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:10.332031 18391 ts_tablet_manager.cc:1905] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:10.332053 18390 raft_consensus.cc:2258] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:10.332543 18390 ts_tablet_manager.cc:1905] T a08432073e084627ba1ebd6f5a74cb48 P 83c7f7a18f994237bd59f4cb2ada9624: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:02:10.333081 17889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39335 (error 108)
I20260929 19:02:10.333320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18094
I20260929 19:02:10.334167 18391 ts_tablet_manager.cc:1918] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:10.334245 18391 log.cc:1196] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/a08432073e084627ba1ebd6f5a74cb48
I20260929 19:02:10.334398 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.334260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39335 (local address 127.0.0.1:40132)
0929 19:02:10.334314 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.334367 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:02:10.334522 18391 ts_tablet_manager.cc:1939] T a08432073e084627ba1ebd6f5a74cb48 P b0e460412b8444fb97b8748018f5a6bc: Deleting consensus metadata
W20260929 19:02:10.334518 17889 catalog_manager.cc:4584] TS 91a4bccd9dc44a59a3536f264fca0cc1 (127.0.98.1:39335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a08432073e084627ba1ebd6f5a74cb48: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39335: connect: Connection refused (error 111)
I20260929 19:02:10.338905 17888 catalog_manager.cc:4841] TS b0e460412b8444fb97b8748018f5a6bc (127.0.98.3:37883): tablet a08432073e084627ba1ebd6f5a74cb48 (table test-table [id=e5f36ff5a22f4e6fbd1f0af5ce0a6f38]) successfully deleted
I20260929 19:02:10.339074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18225
I20260929 19:02:10.339133 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:10.339008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38229 (local address 127.0.0.1:33402)
0929 19:02:10.339071 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:10.339096 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:02:10.343623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 17872
2026-09-29T19:02:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (763 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:10 dist-test-slave-gfsd krb5kdc[18396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:10 dist-test-slave-gfsd krb5kdc[18396](info): set up 2 sockets
Sep 29 19:02:10 dist-test-slave-gfsd krb5kdc[18396](info): commencing operation
krb5kdc: starting...
W20260929 19:02:11.395092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:11.514429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:11.595007 18409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:11.595208 18409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:11.595230 18409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:11.596411 18409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:11.596454 18409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:11.596468 18409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:11.596481 18409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:11.596493 18409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:11.597869 18409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:11.598059 18409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:11.598222 18409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:11.602818 18417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:11.602973 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:11.602996 18409 server_base.cc:1034] running on GCE node
W20260929 19:02:11.603081 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:11.603413 18409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:11.603645 18409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:11.604789 18409 hybrid_clock.cc:648] HybridClock initialized: now 1790708531604763 us; error 43 us; skew 500 ppm
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:11.607427 18409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:11.607890 18409 webserver.cc:460] Webserver started at http://127.0.98.62:40053/ using document root <none> and password file <none>
I20260929 19:02:11.608124 18409 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:11.608191 18409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:11.608332 18409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:11.609452 18409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "736a85e6d65e42468ad8a6255c8fe0a3"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "67429c3ecbad91a8d4c868a2fef68ea8"
server_key_iv: "3cb365cad3478a3940c502ab000d74ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.609817 18409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "736a85e6d65e42468ad8a6255c8fe0a3"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "67429c3ecbad91a8d4c868a2fef68ea8"
server_key_iv: "3cb365cad3478a3940c502ab000d74ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.611250 18409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:11.612171 18424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.612375 18409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:11.612493 18409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "736a85e6d65e42468ad8a6255c8fe0a3"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "67429c3ecbad91a8d4c868a2fef68ea8"
server_key_iv: "3cb365cad3478a3940c502ab000d74ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.612582 18409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:11.622179 18409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:11.624188 18409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:11.624346 18409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:11.628281 18409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37957
I20260929 19:02:11.628310 18476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37957 every 8 connection(s)
I20260929 19:02:11.628666 18409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:11.628947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18409
I20260929 19:02:11.629097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:11.629307 392 external_mini_cluster.cc:1428] Setting key 4d68b614e187bb82fee24288d4dca482
I20260929 19:02:11.629645 18477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:11.632966 18477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: Bootstrap starting.
I20260929 19:02:11.633544 18477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:11.633872 18477 log.cc:824] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:11.634629 18477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: No bootstrap required, opened a new log
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:11.637171 18477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260929 19:02:11.637348 18477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:11.637405 18477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736a85e6d65e42468ad8a6255c8fe0a3, State: Initialized, Role: FOLLOWER
I20260929 19:02:11.637501 18477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260929 19:02:11.637550 18477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:11.637614 18477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:11.637678 18477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:11.638423 18477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260929 19:02:11.638542 18477 leader_election.cc:302] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736a85e6d65e42468ad8a6255c8fe0a3; no voters:
I20260929 19:02:11.638702 18477 leader_election.cc:288] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:11.638772 18482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:11.638948 18482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 736a85e6d65e42468ad8a6255c8fe0a3, State: Running, Role: LEADER
I20260929 19:02:11.638954 18477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:11.639065 18482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260929 19:02:11.639402 18482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736a85e6d65e42468ad8a6255c8fe0a3. Latest consensus state: current_term: 1 leader_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } } }
I20260929 19:02:11.639412 18483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736a85e6d65e42468ad8a6255c8fe0a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } } }
I20260929 19:02:11.639495 18482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:11.639531 18483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:11.639935 18489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:11.640476 18489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:11.641544 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:11.629946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.0.98.62:37957)
0929 19:02:11.630156 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:11.630164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:11.630196 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:02:11.630765 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:11.630774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:11.630835 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:11.630983 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:11.631682 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.633012 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.633822 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.634429 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.634520 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:11.637096 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:11.637101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:11.637101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:11.637113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:11.639190 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.639649 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.639653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.639654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.639771 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.640064 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.640065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.640065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.640378 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:11.640552 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:11.641205 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 19:02:11.641338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260929 19:02:11.642243 18489 catalog_manager.cc:1347] Generated new cluster ID: 5f334fe0962b4d31aa9474a805918a9e
I20260929 19:02:11.642306 18489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:11.649792 18489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:11.650437 18489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:11.655984 18489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: Generated new TSK 0
I20260929 19:02:11.656168 18489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:02:11.686236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:11.769891 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:11.770093 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:11.770115 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:11.771323 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:11.771363 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:11.771389 18503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:11.772861 18503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:11.773101 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:11.773345 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:02:11.779220 18503 server_base.cc:1034] running on GCE node
W20260929 19:02:11.779239 18509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:11.779392 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:11.779529 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:11.779739 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:11.779980 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:11.781147 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790708531781133 us; error 29 us; skew 500 ppm
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:11.783735 18503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:02:11.784207 18503 webserver.cc:460] Webserver started at http://127.0.98.1:44713/ using document root <none> and password file <none>
I20260929 19:02:11.784428 18503 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:11.784476 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:11.784602 18503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:11.785662 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "2928f7ae9af74af599e3af07805dd1ef"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "3bacfbb2110f4de943d4d3a4689dd696"
server_key_iv: "d90f8ebeb428aec9f461ded9beee5a94"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.786017 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2928f7ae9af74af599e3af07805dd1ef"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "3bacfbb2110f4de943d4d3a4689dd696"
server_key_iv: "d90f8ebeb428aec9f461ded9beee5a94"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.787382 18503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:11.788103 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.788311 18503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:11.788424 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "2928f7ae9af74af599e3af07805dd1ef"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "3bacfbb2110f4de943d4d3a4689dd696"
server_key_iv: "d90f8ebeb428aec9f461ded9beee5a94"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.788487 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:11.801074 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:11.803151 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:11.803261 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:11.803519 18503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:11.803761 18503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:11.803790 18503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.803812 18503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:11.803823 18503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.809746 18503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43263
I20260929 19:02:11.809835 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43263 every 8 connection(s)
I20260929 19:02:11.810135 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:11.811533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18503
I20260929 19:02:11.811664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:11.811910 392 external_mini_cluster.cc:1428] Setting key 1186d1983b2567c369fef98e42b7fcbc
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:02:11.821223 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:11.810862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40229 (local address 127.0.98.62:37957)
0929 19:02:11.810999 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:11.811001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:11.811111 (+ 110us) server_negotiation.cc:407] Connection header received
0929 19:02:11.812429 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:11.812430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:11.812441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:11.812510 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:11.813407 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.814052 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.815148 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.815454 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.815543 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:11.818098 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:11.818100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:11.818100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:11.818109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:11.819779 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.820199 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.820200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.820200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.820259 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.820509 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.820509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.820510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.820656 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:11.820733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:11.821087 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 19:02:11.821158 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 19:02:11.821384 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:11.810825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37957 (local address 127.0.98.1:40229)
0929 19:02:11.810996 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:11.811032 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:11.812318 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:11.812598 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:11.812602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:11.812991 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:11.813273 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:11.813279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.814165 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.814166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:11.815043 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:11.815046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.815555 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.815556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:11.815685 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:11.816270 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:11.816295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:11.818001 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:11.819888 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:11.819896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:11.819897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:11.820113 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:11.820346 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:11.820346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:11.820346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:11.820443 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:11.820766 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:11.820773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:11.820942 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 19:02:11.821049 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:02:11.821169 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260929 19:02:11.821970 18632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37957
I20260929 19:02:11.822079 18632 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:11.822255 18632 heartbeater.cc:511] Master 127.0.98.62:37957 requested a full tablet report, sending...
I20260929 19:02:11.822816 18441 ts_manager.cc:194] Registered new tserver with Master: 2928f7ae9af74af599e3af07805dd1ef (127.0.98.1:43263)
I20260929 19:02:11.823556 18441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:02:11.837949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:11.917954 18637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:11.918175 18637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:11.918211 18637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:11.919382 18637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:11.919452 18637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:11.919497 18637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:11.920985 18637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:11.921249 18637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:11.921465 18637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:11.926388 18645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:11.926424 18643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:11.926515 18637 server_base.cc:1034] running on GCE node
W20260929 19:02:11.926610 18642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:11.926827 18637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:11.927047 18637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:11.928232 18637 hybrid_clock.cc:648] HybridClock initialized: now 1790708531928217 us; error 38 us; skew 500 ppm
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:11.930680 18637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:02:11.931119 18637 webserver.cc:460] Webserver started at http://127.0.98.2:39481/ using document root <none> and password file <none>
I20260929 19:02:11.931398 18637 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:11.931448 18637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:11.931578 18637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:11.932579 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "9411edd5938e4b2c990946910b87268c"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "89d8a5ce3bf4ef3c88e74399a154c140"
server_key_iv: "27169dc77ee2bebd0403ed63e88f50a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.932928 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9411edd5938e4b2c990946910b87268c"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "89d8a5ce3bf4ef3c88e74399a154c140"
server_key_iv: "27169dc77ee2bebd0403ed63e88f50a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.934352 18637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:11.935323 18652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.935595 18637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:11.935699 18637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "9411edd5938e4b2c990946910b87268c"
format_stamp: "Formatted at 2026-09-29 19:02:11 on dist-test-slave-gfsd"
server_key: "89d8a5ce3bf4ef3c88e74399a154c140"
server_key_iv: "27169dc77ee2bebd0403ed63e88f50a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:11.935758 18637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:11.946359 18637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:11.948719 18637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:11.948830 18637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:11.949019 18637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:11.949321 18637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:11.949365 18637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.949388 18637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:11.949400 18637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:11.955240 18637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39393
I20260929 19:02:11.955344 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39393 every 8 connection(s)
I20260929 19:02:11.955690 18637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:02:11 dist-test-slave-gfsd krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:11.963054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18637
I20260929 19:02:11.963200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:11.963455 392 external_mini_cluster.cc:1428] Setting key a3f28fe411dec516a2cd69b38b7eeb6a
I20260929 19:02:11.966053 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:11.956633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56581 (local address 127.0.98.62:37957)
0929 19:02:11.956692 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:11.956695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:11.956900 (+ 205us) server_negotiation.cc:407] Connection header received
0929 19:02:11.957829 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:11.957831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:11.957841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:11.957985 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:11.958605 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.959120 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.960283 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.960467 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.960524 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:11.962697 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:11.962698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:11.962698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:11.962705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:11.964426 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.964963 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.964964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.964964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.965017 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:11.965308 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:11.965308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:11.965309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:11.965422 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:11.965479 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:11.965831 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 19:02:11.965933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:02:11.966078 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:11.956593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37957 (local address 127.0.98.2:56581)
0929 19:02:11.956756 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:11.956797 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:11.957701 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:11.958056 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:11.958059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:11.958320 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:11.958522 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:11.958527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.959217 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.959218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:11.960209 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:11.960212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:11.960539 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:11.960540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:11.960636 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:11.961107 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:11.961115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:11.962580 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:11.964506 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:11.964512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:11.964514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:11.964840 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:11.965083 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:11.965084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:11.965084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:11.965234 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:11.965490 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:11.965496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:11.965670 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 19:02:11.965791 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 19:02:11.965921 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260929 19:02:11.966652 18766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37957
I20260929 19:02:11.966746 18766 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:11.966886 18766 heartbeater.cc:511] Master 127.0.98.62:37957 requested a full tablet report, sending...
I20260929 19:02:11.967324 18441 ts_manager.cc:194] Registered new tserver with Master: 9411edd5938e4b2c990946910b87268c (127.0.98.2:39393)
I20260929 19:02:11.967852 18441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56581
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:02:11.990516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:12.072346 18771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:12.072544 18771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:12.072568 18771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:12.073650 18771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:12.073683 18771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:12.073709 18771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:12.075183 18771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:12.075374 18771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:12.075536 18771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:12.080523 18779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.080540 18776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.080597 18777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.080600 18771 server_base.cc:1034] running on GCE node
I20260929 19:02:12.080966 18771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:12.081198 18771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:12.082363 18771 hybrid_clock.cc:648] HybridClock initialized: now 1790708532082340 us; error 37 us; skew 500 ppm
Sep 29 19:02:12 dist-test-slave-gfsd krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:12.085094 18771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:02:12.085521 18771 webserver.cc:460] Webserver started at http://127.0.98.3:36339/ using document root <none> and password file <none>
I20260929 19:02:12.085752 18771 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:12.085795 18771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:12.085949 18771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:12.086983 18771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce3dc5d12ad742818fac4306a165ca13"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "c754d2b777c83b13337fc2491a6abd95"
server_key_iv: "cc29531e9cc8a7a74ef00ab1defd3f41"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.087396 18771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce3dc5d12ad742818fac4306a165ca13"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "c754d2b777c83b13337fc2491a6abd95"
server_key_iv: "cc29531e9cc8a7a74ef00ab1defd3f41"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.088716 18771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:12.089500 18786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.089664 18771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:12.089753 18771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "ce3dc5d12ad742818fac4306a165ca13"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "c754d2b777c83b13337fc2491a6abd95"
server_key_iv: "cc29531e9cc8a7a74ef00ab1defd3f41"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.089807 18771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:12.102982 18771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:12.104962 18771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:12.105077 18771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:12.105245 18771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:12.105551 18771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:12.105590 18771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.105633 18771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:12.105661 18771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.111382 18771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32961
I20260929 19:02:12.111428 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32961 every 8 connection(s)
I20260929 19:02:12.111739 18771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:12.115504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18771
I20260929 19:02:12.115610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:12.115803 392 external_mini_cluster.cc:1428] Setting key ed7ef89d5de211391955e863304097bf
Sep 29 19:02:12 dist-test-slave-gfsd krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:12.121811 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.112607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33081 (local address 127.0.98.62:37957)
0929 19:02:12.112675 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.112676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.112889 (+ 213us) server_negotiation.cc:407] Connection header received
0929 19:02:12.113877 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.113878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.113892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.113947 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.114541 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.115146 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.116683 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.116942 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.117001 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.119025 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.119027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.119027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:12.119033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.120580 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:12.120961 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:12.120962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:12.120963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:12.121023 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:12.121245 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:12.121246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:12.121246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:12.121365 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.121426 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.121660 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.121730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:02:12.121945 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.112580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37957 (local address 127.0.98.3:33081)
0929 19:02:12.112769 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.112803 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.113757 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.114020 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.114024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.114261 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:12.114447 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.114451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.115237 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.115238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.116522 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.116526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.117018 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.117019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.117120 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.117555 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:12.117563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:12.118937 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:12.120661 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:12.120667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:12.120669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:12.120878 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:12.121090 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:12.121091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:12.121091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:12.121184 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:12.121425 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:12.121429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:12.121536 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.121618 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.121730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260929 19:02:12.122434 18900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37957
I20260929 19:02:12.122515 18900 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:12.122712 18900 heartbeater.cc:511] Master 127.0.98.62:37957 requested a full tablet report, sending...
I20260929 19:02:12.123167 18441 ts_manager.cc:194] Registered new tserver with Master: ce3dc5d12ad742818fac4306a165ca13 (127.0.98.3:32961)
I20260929 19:02:12.123564 18441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33081
I20260929 19:02:12.127830 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:12.135655 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.129129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.0.98.62:37957)
0929 19:02:12.129201 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.129203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.129314 (+ 111us) server_negotiation.cc:407] Connection header received
0929 19:02:12.129388 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.129389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.129399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.129458 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.130037 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.130502 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.131249 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.131567 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.131655 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.132889 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.132891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.132891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:12.132898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.134275 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:12.134702 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:12.134703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:12.134704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:12.134759 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:12.135003 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:12.135004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:12.135040 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:12.135179 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.135340 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.135490 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.135558 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:02:12.137123 18441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:12.138859 18441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:12.145464 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.98.3:32961)
0929 19:02:12.141830 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.141834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.141843 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:12.141884 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.141886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.141901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.142135 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.142205 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.142899 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.144549 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145045 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145140 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145205 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.145344 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.145382 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":81,"threads_started":1}
I20260929 19:02:12.145457 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32961 (local address 127.0.0.1:41472)
0929 19:02:12.141263 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.141303 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.141386 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.142012 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.142013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.142015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.142062 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.142063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.143004 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.143005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.144439 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.144442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145139 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.145228 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145243 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.145312 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.145378 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":48,"thread_start_us":87,"threads_started":1}
I20260929 19:02:12.145610 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39393 (local address 127.0.0.1:49310)
0929 19:02:12.141841 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.141853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.141917 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.142193 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.142194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.142196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.142228 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.142231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.143414 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.143415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.144347 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.144350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145337 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.145408 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145428 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.145474 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.145526 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":7,"thread_start_us":45,"threads_started":1}
I20260929 19:02:12.145707 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.98.1:43263)
0929 19:02:12.141566 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.141570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.141582 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:02:12.141624 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.141626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.141644 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.141730 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.141915 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.142972 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.144525 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145186 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145245 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145300 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.145572 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.145622 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260929 19:02:12.145828 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.0.98.2:39393)
0929 19:02:12.141740 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.141746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.141977 (+ 231us) server_negotiation.cc:407] Connection header received
0929 19:02:12.142017 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.142019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.142033 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.142105 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.142329 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.143327 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.144435 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145163 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145247 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145300 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.145697 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.145756 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260929 19:02:12.145958 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.141138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43263 (local address 127.0.0.1:48008)
0929 19:02:12.141478 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.141489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.141541 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.141757 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.141760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.141762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.141826 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.141828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.143060 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.143061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.144443 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.144445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.145345 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.145346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.145408 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.145412 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.145467 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.145526 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":77,"thread_start_us":46,"threads_started":1}
I20260929 19:02:12.146554 18700 tablet_service.cc:1467] Processing CreateTablet for tablet 73f652d13ba242639727351c188f1164 (DEFAULT_TABLE table=test-table [id=260a539ae5b44fc885d6cfb65ec42bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.146566 18566 tablet_service.cc:1467] Processing CreateTablet for tablet 73f652d13ba242639727351c188f1164 (DEFAULT_TABLE table=test-table [id=260a539ae5b44fc885d6cfb65ec42bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.146919 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f652d13ba242639727351c188f1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.146919 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f652d13ba242639727351c188f1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.150349 18915 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: Bootstrap starting.
I20260929 19:02:12.150993 18915 tablet_bootstrap.cc:654] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.150945 18834 tablet_service.cc:1467] Processing CreateTablet for tablet 73f652d13ba242639727351c188f1164 (DEFAULT_TABLE table=test-table [id=260a539ae5b44fc885d6cfb65ec42bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.151247 18834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f652d13ba242639727351c188f1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.151444 18914 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Bootstrap starting.
I20260929 19:02:12.151599 18915 log.cc:824] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.152067 18914 tablet_bootstrap.cc:654] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.152415 18915 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: No bootstrap required, opened a new log
I20260929 19:02:12.152484 18915 ts_tablet_manager.cc:1397] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:02:12.152607 18914 log.cc:824] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.153375 18914 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: No bootstrap required, opened a new log
I20260929 19:02:12.153448 18914 ts_tablet_manager.cc:1397] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:12.154366 18918 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Bootstrap starting.
I20260929 19:02:12.154551 18915 raft_consensus.cc:348] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.154733 18915 raft_consensus.cc:374] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.154771 18915 raft_consensus.cc:729] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2928f7ae9af74af599e3af07805dd1ef, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.154910 18915 consensus_queue.cc:260] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.155059 18918 tablet_bootstrap.cc:654] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.155202 18915 ts_tablet_manager.cc:1428] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:02:12.155421 18632 heartbeater.cc:503] Master 127.0.98.62:37957 was elected leader, sending a full tablet report...
I20260929 19:02:12.155495 18914 raft_consensus.cc:348] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.155618 18914 raft_consensus.cc:374] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.155639 18914 raft_consensus.cc:729] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9411edd5938e4b2c990946910b87268c, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.155644 18918 log.cc:824] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.155767 18914 consensus_queue.cc:260] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.156075 18914 ts_tablet_manager.cc:1428] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:12.156090 18766 heartbeater.cc:503] Master 127.0.98.62:37957 was elected leader, sending a full tablet report...
I20260929 19:02:12.156905 18918 tablet_bootstrap.cc:492] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: No bootstrap required, opened a new log
I20260929 19:02:12.156984 18918 ts_tablet_manager.cc:1397] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:02:12.158315 18918 raft_consensus.cc:348] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.158481 18918 raft_consensus.cc:374] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.158507 18918 raft_consensus.cc:729] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3dc5d12ad742818fac4306a165ca13, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.158622 18918 consensus_queue.cc:260] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.158895 18918 ts_tablet_manager.cc:1428] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:12.159011 18900 heartbeater.cc:503] Master 127.0.98.62:37957 was elected leader, sending a full tablet report...
W20260929 19:02:12.206298 18767 tablet.cc:2369] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:12.238405 18921 raft_consensus.cc:482] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:12.238574 18921 raft_consensus.cc:504] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.239125 18921 leader_election.cc:288] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce3dc5d12ad742818fac4306a165ca13 (127.0.98.3:32961), 2928f7ae9af74af599e3af07805dd1ef (127.0.98.1:43263)
I20260929 19:02:12.243062 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.239542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58521 (local address 127.0.98.1:43263)
0929 19:02:12.239588 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.239590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.239598 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:12.239625 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.239626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.239636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.239709 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.239877 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.240595 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242129 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.242690 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.242778 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.242953 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.242992 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:02:12.243062 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.239371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53475 (local address 127.0.98.3:32961)
0929 19:02:12.239426 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.239429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.239463 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:02:12.239494 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.239495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.239505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.239562 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.239742 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.240596 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242128 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.242690 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.242777 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.242951 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.242992 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:02:12.243085 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.239320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32961 (local address 127.0.98.2:53475)
0929 19:02:12.239371 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.239383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.239424 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.239578 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.239579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.239581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.239650 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.239652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.240690 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.240691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.242019 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.242023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.242786 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.242869 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.242874 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.242933 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.242980 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":118}
I20260929 19:02:12.243093 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.239327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43263 (local address 127.0.98.2:58521)
0929 19:02:12.239501 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.239510 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.239577 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.239711 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.239711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.239713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.239801 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.239803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.240690 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.240691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.242019 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.242023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.242787 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.242787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.242869 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.242874 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.242933 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.242980 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":50,"thread_start_us":114,"threads_started":1}
I20260929 19:02:12.243489 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f652d13ba242639727351c188f1164" candidate_uuid: "9411edd5938e4b2c990946910b87268c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3dc5d12ad742818fac4306a165ca13" is_pre_election: true
I20260929 19:02:12.243522 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f652d13ba242639727351c188f1164" candidate_uuid: "9411edd5938e4b2c990946910b87268c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2928f7ae9af74af599e3af07805dd1ef" is_pre_election: true
I20260929 19:02:12.243659 18854 raft_consensus.cc:2454] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9411edd5938e4b2c990946910b87268c in term 0.
I20260929 19:02:12.243677 18586 raft_consensus.cc:2454] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9411edd5938e4b2c990946910b87268c in term 0.
I20260929 19:02:12.243880 18654 leader_election.cc:302] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2928f7ae9af74af599e3af07805dd1ef, 9411edd5938e4b2c990946910b87268c; no voters:
I20260929 19:02:12.244024 18921 raft_consensus.cc:2781] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:12.244120 18921 raft_consensus.cc:482] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:12.244154 18921 raft_consensus.cc:3037] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:12.244827 18921 raft_consensus.cc:504] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.244972 18921 leader_election.cc:288] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [CANDIDATE]: Term 1 election: Requested vote from peers ce3dc5d12ad742818fac4306a165ca13 (127.0.98.3:32961), 2928f7ae9af74af599e3af07805dd1ef (127.0.98.1:43263)
I20260929 19:02:12.245106 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f652d13ba242639727351c188f1164" candidate_uuid: "9411edd5938e4b2c990946910b87268c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2928f7ae9af74af599e3af07805dd1ef"
I20260929 19:02:12.245169 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f652d13ba242639727351c188f1164" candidate_uuid: "9411edd5938e4b2c990946910b87268c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3dc5d12ad742818fac4306a165ca13"
I20260929 19:02:12.245190 18586 raft_consensus.cc:3037] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:12.245270 18854 raft_consensus.cc:3037] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:12.245949 18586 raft_consensus.cc:2454] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9411edd5938e4b2c990946910b87268c in term 1.
I20260929 19:02:12.246116 18654 leader_election.cc:302] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2928f7ae9af74af599e3af07805dd1ef, 9411edd5938e4b2c990946910b87268c; no voters:
I20260929 19:02:12.246145 18854 raft_consensus.cc:2454] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9411edd5938e4b2c990946910b87268c in term 1.
I20260929 19:02:12.246241 18921 raft_consensus.cc:2781] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:12.246407 18921 raft_consensus.cc:686] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 1 LEADER]: Becoming Leader. State: Replica: 9411edd5938e4b2c990946910b87268c, State: Running, Role: LEADER
I20260929 19:02:12.246522 18921 consensus_queue.cc:237] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } }
I20260929 19:02:12.247509 18440 catalog_manager.cc:5495] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9411edd5938e4b2c990946910b87268c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9411edd5938e4b2c990946910b87268c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9411edd5938e4b2c990946910b87268c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:12.261286 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.258793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.0.98.2:39393)
0929 19:02:12.258848 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.258850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.258898 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:02:12.258979 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.258980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.258997 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.259055 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:12.259171 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.259693 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.260495 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.260673 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.260727 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.260851 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:12.260940 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:12.261010 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.261135 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.261165 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:02:12.263046 18586 raft_consensus.cc:1261] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 1 FOLLOWER]: Refusing update from remote peer 9411edd5938e4b2c990946910b87268c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:12.263160 18854 raft_consensus.cc:1261] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 1 FOLLOWER]: Refusing update from remote peer 9411edd5938e4b2c990946910b87268c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:12.263434 18921 consensus_queue.cc:1035] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2928f7ae9af74af599e3af07805dd1ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:12.263574 18924 consensus_queue.cc:1035] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce3dc5d12ad742818fac4306a165ca13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:12.268231 18441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:12.268349 18441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:12.269111 18441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 736a85e6d65e42468ad8a6255c8fe0a3: Sending DeleteTablet for 3 replicas of tablet 73f652d13ba242639727351c188f1164
I20260929 19:02:12.269450 18700 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f652d13ba242639727351c188f1164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49310
I20260929 19:02:12.269438 18834 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f652d13ba242639727351c188f1164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260929 19:02:12.269477 18566 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f652d13ba242639727351c188f1164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260929 19:02:12.269712 18939 tablet_replica.cc:330] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef: stopping tablet replica
I20260929 19:02:12.269716 18938 tablet_replica.cc:330] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: stopping tablet replica
I20260929 19:02:12.269831 18939 raft_consensus.cc:2229] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:12.269847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18503
I20260929 19:02:12.269932 18938 raft_consensus.cc:2229] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:12.269948 18939 raft_consensus.cc:2258] T 73f652d13ba242639727351c188f1164 P 2928f7ae9af74af599e3af07805dd1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:12.271037 18938 raft_consensus.cc:2258] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:12.271584 18938 ts_tablet_manager.cc:1905] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:12.273546 18938 ts_tablet_manager.cc:1918] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:12.273664 18938 log.cc:1196] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/73f652d13ba242639727351c188f1164
I20260929 19:02:12.274019 18938 ts_tablet_manager.cc:1939] T 73f652d13ba242639727351c188f1164 P ce3dc5d12ad742818fac4306a165ca13: Deleting consensus metadata
I20260929 19:02:12.274423 18937 tablet_replica.cc:330] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: stopping tablet replica
I20260929 19:02:12.274524 18428 catalog_manager.cc:4841] TS ce3dc5d12ad742818fac4306a165ca13 (127.0.98.3:32961): tablet 73f652d13ba242639727351c188f1164 (table test-table [id=260a539ae5b44fc885d6cfb65ec42bb7]) successfully deleted
I20260929 19:02:12.274570 18937 raft_consensus.cc:2229] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:12.274749 18937 raft_consensus.cc:2258] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:12.275578 18937 ts_tablet_manager.cc:1905] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:02:12.277390 18426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43263 (error 108)
I20260929 19:02:12.277565 18937 ts_tablet_manager.cc:1918] T 73f652d13ba242639727351c188f1164 P 9411edd5938e4b2c990946910b87268c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:12.277628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18637
I20260929 19:02:12.278555 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.278415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43263 (local address 127.0.0.1:48014)
0929 19:02:12.278475 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.278514 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 19:02:12.278712 18426 catalog_manager.cc:4584] TS 2928f7ae9af74af599e3af07805dd1ef (127.0.98.1:43263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73f652d13ba242639727351c188f1164: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43263: connect: Connection refused (error 111)
I20260929 19:02:12.282732 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.282611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39393 (local address 127.0.0.1:49330)
0929 19:02:12.282667 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.282702 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:02:12.282745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18771
I20260929 19:02:12.287551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18409
2026-09-29T19:02:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T19:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:12Z Disabled control of system clock
I20260929 19:02:12.314816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:12.394434 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:12.394659 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:12.394680 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:12.395819 18944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:12.395857 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:12.395869 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:12.395882 18944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:12.395893 18944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:12.397181 18944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:12.397375 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:12.397629 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:12.402431 18950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.402511 18949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.402486 18952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.402658 18944 server_base.cc:1034] running on GCE node
I20260929 19:02:12.402927 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:12.403216 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:12.404392 18944 hybrid_clock.cc:648] HybridClock initialized: now 1790708532404369 us; error 41 us; skew 500 ppm
I20260929 19:02:12.410781 18944 webserver.cc:460] Webserver started at http://127.0.98.62:44537/ using document root <none> and password file <none>
I20260929 19:02:12.411046 18944 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:12.411118 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:12.411263 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:12.412446 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "74b73709ba774b05939f39f161a3d159"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6e7382be630975f31d430361937b4a68"
server_key_iv: "447042ca5d52375989fad65a33becec0"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.412810 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "74b73709ba774b05939f39f161a3d159"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6e7382be630975f31d430361937b4a68"
server_key_iv: "447042ca5d52375989fad65a33becec0"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.414155 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:12.415046 18958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.415241 18944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:12.415370 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "74b73709ba774b05939f39f161a3d159"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6e7382be630975f31d430361937b4a68"
server_key_iv: "447042ca5d52375989fad65a33becec0"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.415444 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:12.435374 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:12.435706 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:12.435861 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:12.439702 18944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33919
I20260929 19:02:12.439735 19010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33919 every 8 connection(s)
I20260929 19:02:12.440090 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:12.441057 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.443257 19011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: Bootstrap starting.
I20260929 19:02:12.443881 19011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.444341 19011 log.cc:824] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.445209 19011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: No bootstrap required, opened a new log
I20260929 19:02:12.446987 19011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260929 19:02:12.447134 19011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.447160 19011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b73709ba774b05939f39f161a3d159, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.447399 19011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260929 19:02:12.447496 19011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:12.447554 19011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:12.447610 19011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:12.448180 19011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260929 19:02:12.448272 19011 leader_election.cc:302] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b73709ba774b05939f39f161a3d159; no voters:
I20260929 19:02:12.448498 19011 leader_election.cc:288] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:12.448578 19014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:12.448751 19014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [term 1 LEADER]: Becoming Leader. State: Replica: 74b73709ba774b05939f39f161a3d159, State: Running, Role: LEADER
I20260929 19:02:12.448768 19011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:12.448851 19014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260929 19:02:12.449165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 18944
I20260929 19:02:12.449283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:12.449532 392 external_mini_cluster.cc:1428] Setting key 4459a89449235fd93769294bb9516042
I20260929 19:02:12.450254 19016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b73709ba774b05939f39f161a3d159. Latest consensus state: current_term: 1 leader_uuid: "74b73709ba774b05939f39f161a3d159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260929 19:02:12.450332 19016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:12.450276 19015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b73709ba774b05939f39f161a3d159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b73709ba774b05939f39f161a3d159" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260929 19:02:12.450378 19015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159 [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:12.451161 19030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:12.451218 19030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:12.451272 19029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:12.451463 19029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:12.452934 19029 catalog_manager.cc:1347] Generated new cluster ID: a130ad44b605459c9d8245580d2562e5
I20260929 19:02:12.452986 19029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:12.455716 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.450604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.0.98.62:33919)
0929 19:02:12.450892 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.450902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.450937 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:02:12.451045 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.451050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.451103 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.451254 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.451534 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.452429 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.453523 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.454149 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.454288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.454975 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.454979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.454979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:12.455192 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.455232 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:12.455247 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.455342 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.455422 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.455499 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":92,"thread_start_us":73,"threads_started":1}
I20260929 19:02:12.458581 19029 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:12.459254 19029 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:12.478962 19029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: Generated new TSK 0
I20260929 19:02:12.479192 19029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:12.481360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:12.581555 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:12.581768 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:12.581790 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:12.582986 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:12.583035 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:12.583062 19035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:12.584439 19035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:12.584687 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:12.585019 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:12.590476 19041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.590484 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.590484 19040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.590804 19035 server_base.cc:1034] running on GCE node
I20260929 19:02:12.591022 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:12.591279 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:12.592473 19035 hybrid_clock.cc:648] HybridClock initialized: now 1790708532592460 us; error 31 us; skew 500 ppm
I20260929 19:02:12.593783 19035 webserver.cc:460] Webserver started at http://127.0.98.1:33251/ using document root <none> and password file <none>
I20260929 19:02:12.594095 19035 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:12.594172 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:12.594306 19035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:12.595463 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "a765888d8de24b829d7bf02ab268994a"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "37d4e18d4c780d011017f8b980f4e300"
server_key_iv: "dcc05c8bca0859f6224dd55f5cbe6bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.595825 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a765888d8de24b829d7bf02ab268994a"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "37d4e18d4c780d011017f8b980f4e300"
server_key_iv: "dcc05c8bca0859f6224dd55f5cbe6bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.597219 19035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:02:12.597971 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.598224 19035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:12.598315 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "a765888d8de24b829d7bf02ab268994a"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "37d4e18d4c780d011017f8b980f4e300"
server_key_iv: "dcc05c8bca0859f6224dd55f5cbe6bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.598387 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:12.609750 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:12.610059 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:12.610189 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:12.610402 19035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:12.610750 19035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:12.610791 19035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.610818 19035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:12.610841 19035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.616914 19035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38273
I20260929 19:02:12.616971 19162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38273 every 8 connection(s)
I20260929 19:02:12.617321 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:12.622632 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.618104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48165 (local address 127.0.98.62:33919)
0929 19:02:12.618149 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.618151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.618366 (+ 215us) server_negotiation.cc:407] Connection header received
0929 19:02:12.618855 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.618856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.618868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.618963 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.619771 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.620250 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.621259 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.621456 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.621522 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.622354 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.622355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.622355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:12.622359 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.622392 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:12.622396 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.622437 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.622544 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.622572 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:12.622748 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.618021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.1:48165)
0929 19:02:12.618247 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.618280 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.618772 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.619022 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.619026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.619033 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:12.619668 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.619674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.620334 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.620335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.621179 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.621181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.621536 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.621537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.621640 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.622224 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:12.622225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:12.622254 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:12.622276 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:12.622446 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:12.622449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:12.622451 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.622497 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.622573 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
I20260929 19:02:12.623400 19163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260929 19:02:12.623482 19163 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:12.623661 19163 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260929 19:02:12.624151 18975 ts_manager.cc:194] Registered new tserver with Master: a765888d8de24b829d7bf02ab268994a (127.0.98.1:38273)
I20260929 19:02:12.624796 18975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48165
I20260929 19:02:12.627115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19035
I20260929 19:02:12.627200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:12.627431 392 external_mini_cluster.cc:1428] Setting key 1dfecba76652272b3a3dd293aadec92a
I20260929 19:02:12.628328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:12.708879 19166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:12.709138 19166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:12.709160 19166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:12.710258 19166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:12.710294 19166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:12.710317 19166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:12.711790 19166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:12.711987 19166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:12.712221 19166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:12.717478 19172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.717486 19174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.717630 19166 server_base.cc:1034] running on GCE node
W20260929 19:02:12.717510 19171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.717887 19166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:12.718132 19166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:12.719336 19166 hybrid_clock.cc:648] HybridClock initialized: now 1790708532719302 us; error 48 us; skew 500 ppm
I20260929 19:02:12.720569 19166 webserver.cc:460] Webserver started at http://127.0.98.2:45469/ using document root <none> and password file <none>
I20260929 19:02:12.720819 19166 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:12.720888 19166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:12.721027 19166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:12.722088 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a3872008b954ee8aeebe43ec43393ed"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6a4cb0625a2b3a0e8bcf77adfce20fcf"
server_key_iv: "5b1512a6db9e3a736cd3cedf8104e9aa"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.722448 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a3872008b954ee8aeebe43ec43393ed"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6a4cb0625a2b3a0e8bcf77adfce20fcf"
server_key_iv: "5b1512a6db9e3a736cd3cedf8104e9aa"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.723829 19166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:12.724532 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.724704 19166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:12.724799 19166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "7a3872008b954ee8aeebe43ec43393ed"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "6a4cb0625a2b3a0e8bcf77adfce20fcf"
server_key_iv: "5b1512a6db9e3a736cd3cedf8104e9aa"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.724879 19166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:12.758788 19166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:12.759140 19166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:12.759331 19166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:12.759591 19166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:12.759927 19166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:12.759984 19166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.760020 19166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:12.760066 19166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.765805 19166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41085
I20260929 19:02:12.765851 19293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41085 every 8 connection(s)
I20260929 19:02:12.766202 19166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:12.771808 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.767021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55785 (local address 127.0.98.62:33919)
0929 19:02:12.767099 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.767101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.767406 (+ 305us) server_negotiation.cc:407] Connection header received
0929 19:02:12.767953 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.767954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.767965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.768029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.768743 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.769229 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.770358 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.770543 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.770610 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.771566 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.771567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.771568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:12.771572 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.771585 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:12.771589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.771633 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.771722 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.771747 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 19:02:12.772159 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.766977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.2:55785)
0929 19:02:12.767215 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.767265 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.767850 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.768088 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.768092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.768101 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:12.768668 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.768672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.769326 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.769327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.770274 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.770277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.770641 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.770642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.770769 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.771421 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:12.771422 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:12.771454 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:12.771481 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:12.771649 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:12.771654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:12.771657 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.771714 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.771800 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260929 19:02:12.772751 19294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260929 19:02:12.772939 19294 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:12.773159 19294 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260929 19:02:12.773622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19166
I20260929 19:02:12.773638 18975 ts_manager.cc:194] Registered new tserver with Master: 7a3872008b954ee8aeebe43ec43393ed (127.0.98.2:41085)
I20260929 19:02:12.773729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:12.773967 392 external_mini_cluster.cc:1428] Setting key 40669a4870011024a1e55d87d6c825e5
I20260929 19:02:12.774127 18975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55785
I20260929 19:02:12.774981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:12.858105 19297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:12.858348 19297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:12.858384 19297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:12.859619 19297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:12.859679 19297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:12.859724 19297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:12.861207 19297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:12.861518 19297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:12.861827 19297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:12.867092 19305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.867092 19302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:12.867259 19303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:12.867488 19297 server_base.cc:1034] running on GCE node
I20260929 19:02:12.867909 19297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:12.868175 19297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:12.869359 19297 hybrid_clock.cc:648] HybridClock initialized: now 1790708532869342 us; error 32 us; skew 500 ppm
I20260929 19:02:12.871186 19297 webserver.cc:460] Webserver started at http://127.0.98.3:43533/ using document root <none> and password file <none>
I20260929 19:02:12.871484 19297 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:12.871560 19297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:12.871727 19297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:12.872805 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "90f3988879864ad593f1a70f1ac91c97"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "b910d797412dd80b4a8191f893cbfbdc"
server_key_iv: "3d34ac84392c179b829a7c6a35990388"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.873193 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "90f3988879864ad593f1a70f1ac91c97"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "b910d797412dd80b4a8191f893cbfbdc"
server_key_iv: "3d34ac84392c179b829a7c6a35990388"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.874552 19297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:12.875445 19311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.875638 19297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:12.875738 19297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "90f3988879864ad593f1a70f1ac91c97"
format_stamp: "Formatted at 2026-09-29 19:02:12 on dist-test-slave-gfsd"
server_key: "b910d797412dd80b4a8191f893cbfbdc"
server_key_iv: "3d34ac84392c179b829a7c6a35990388"
server_key_version: "encryptionkey@0"
I20260929 19:02:12.875905 19297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:12.884299 19297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:12.884619 19297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:12.884769 19297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:12.884996 19297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:12.885366 19297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:12.885427 19297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.885473 19297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:12.885501 19297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:12.891508 19297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44789
I20260929 19:02:12.891582 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44789 every 8 connection(s)
I20260929 19:02:12.891933 19297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:12.897293 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.892689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37415 (local address 127.0.98.62:33919)
0929 19:02:12.892765 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.892767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.893034 (+ 267us) server_negotiation.cc:407] Connection header received
0929 19:02:12.893524 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.893526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.893536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.893591 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.894288 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.894815 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.895905 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.896116 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.896188 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.896996 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.896997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.896997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:12.897002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.897019 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:12.897023 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.897083 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.897203 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.897231 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:02:12.897383 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.892685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.3:37415)
0929 19:02:12.892901 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.892936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.893427 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.893664 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.893667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.893675 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:12.894196 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.894201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.894909 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.894910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.895812 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.895814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.896203 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.896204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.896318 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.896836 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:12.896836 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:12.896865 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:12.896889 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:12.897090 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:12.897094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:12.897096 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.897151 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.897228 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260929 19:02:12.897881 19425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260929 19:02:12.897970 19425 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:12.898125 19425 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260929 19:02:12.898488 18975 ts_manager.cc:194] Registered new tserver with Master: 90f3988879864ad593f1a70f1ac91c97 (127.0.98.3:44789)
I20260929 19:02:12.898862 18975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37415
I20260929 19:02:12.899982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19297
I20260929 19:02:12.900061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:12.900240 392 external_mini_cluster.cc:1428] Setting key 933afdbd6b07f22160abbbd2b9e1d1f6
I20260929 19:02:12.901093 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:12.905504 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.902572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.98.62:33919)
0929 19:02:12.902631 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.902633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.902776 (+ 143us) server_negotiation.cc:407] Connection header received
0929 19:02:12.902838 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.902838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.902849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.902901 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:12.903126 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.903656 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.904367 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.904540 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.904590 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.905270 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:12.905271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:12.905271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:12.905275 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:12.905285 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:12.905288 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:12.905347 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.905389 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.905398 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:02:12.907188 18975 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:12.908844 18975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:12.915989 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.0.98.2:41085)
0929 19:02:12.911686 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.911693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.911861 (+ 168us) server_negotiation.cc:407] Connection header received
0929 19:02:12.911905 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.911907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.911923 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.912020 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.912182 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913353 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.914712 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.915508 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.915592 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.915704 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.915851 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.915898 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260929 19:02:12.916016 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.0.1:38524)
0929 19:02:12.911605 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.911669 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.911742 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.912430 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.912430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.912432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.912475 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.912477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913384 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.913385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.914769 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.914772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.915683 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.915684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.915734 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.915744 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.915839 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.915899 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":120,"thread_start_us":76,"threads_started":1}
I20260929 19:02:12.916016 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41085 (local address 127.0.0.1:37722)
0929 19:02:12.911741 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.911750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.911797 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.912034 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.912037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.912039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.912110 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.912112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913452 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.913453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.914626 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.914628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.915618 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.915619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.915726 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.915742 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.915830 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.915899 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":39,"thread_start_us":110,"threads_started":1}
I20260929 19:02:12.916287 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.0.98.1:38273)
0929 19:02:12.912153 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.912158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.912167 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:12.912210 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.912251 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.912269 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.912343 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.912554 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913282 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.914861 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.915606 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.915671 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.915745 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.916158 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.916202 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":90,"threads_started":1}
I20260929 19:02:12.917021 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44789 (local address 127.0.0.1:47748)
0929 19:02:12.911960 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:12.911969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:12.912029 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:12.912258 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:12.912259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:12.912261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:12.912300 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.912302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913535 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.913536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.914769 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:12.914772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.916777 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.916778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:12.916837 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.916841 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:12.916890 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:02:12.916938 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":14,"thread_start_us":72,"threads_started":1}
I20260929 19:02:12.916955 19228 tablet_service.cc:1467] Processing CreateTablet for tablet a41eeac8021c4945ad0aea2094116f49 (DEFAULT_TABLE table=test-table [id=e769d3d23cef4d468257bb999c19ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.916990 19097 tablet_service.cc:1467] Processing CreateTablet for tablet a41eeac8021c4945ad0aea2094116f49 (DEFAULT_TABLE table=test-table [id=e769d3d23cef4d468257bb999c19ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.917263 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:12.911775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.0.98.3:44789)
0929 19:02:12.911939 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:12.911943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:12.912015 (+ 72us) server_negotiation.cc:407] Connection header received
0929 19:02:12.912103 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:12.912105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:12.912119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:12.912188 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:12.912380 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.913459 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.915791 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:12.916588 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:12.916660 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:12.916743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:12.917141 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 19:02:12.917187 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260929 19:02:12.917341 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41eeac8021c4945ad0aea2094116f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.917330 19228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41eeac8021c4945ad0aea2094116f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.920053 19439 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Bootstrap starting.
I20260929 19:02:12.920078 19359 tablet_service.cc:1467] Processing CreateTablet for tablet a41eeac8021c4945ad0aea2094116f49 (DEFAULT_TABLE table=test-table [id=e769d3d23cef4d468257bb999c19ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:12.920414 19359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41eeac8021c4945ad0aea2094116f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:12.920696 19440 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Bootstrap starting.
I20260929 19:02:12.920830 19439 tablet_bootstrap.cc:654] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.921119 19439 log.cc:824] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.921345 19440 tablet_bootstrap.cc:654] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.921681 19440 log.cc:824] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.921926 19439 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: No bootstrap required, opened a new log
I20260929 19:02:12.921999 19439 ts_tablet_manager.cc:1397] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:12.922431 19440 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: No bootstrap required, opened a new log
I20260929 19:02:12.922580 19440 ts_tablet_manager.cc:1397] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:12.923967 19443 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Bootstrap starting.
I20260929 19:02:12.923914 19439 raft_consensus.cc:348] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.924080 19439 raft_consensus.cc:374] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.924129 19439 raft_consensus.cc:729] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3872008b954ee8aeebe43ec43393ed, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.924258 19439 consensus_queue.cc:260] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.924531 19439 ts_tablet_manager.cc:1428] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:02:12.924605 19443 tablet_bootstrap.cc:654] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:12.924674 19294 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260929 19:02:12.924860 19443 log.cc:824] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:12.925611 19443 tablet_bootstrap.cc:492] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: No bootstrap required, opened a new log
I20260929 19:02:12.925665 19443 ts_tablet_manager.cc:1397] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:12.926043 19440 raft_consensus.cc:348] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.926194 19440 raft_consensus.cc:374] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.926241 19440 raft_consensus.cc:729] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a765888d8de24b829d7bf02ab268994a, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.926374 19440 consensus_queue.cc:260] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.926721 19163 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260929 19:02:12.926707 19440 ts_tablet_manager.cc:1428] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 19:02:12.928056 19443 raft_consensus.cc:348] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.928172 19443 raft_consensus.cc:374] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:12.928192 19443 raft_consensus.cc:729] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f3988879864ad593f1a70f1ac91c97, State: Initialized, Role: FOLLOWER
I20260929 19:02:12.928313 19443 consensus_queue.cc:260] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:12.928571 19443 ts_tablet_manager.cc:1428] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:12.928591 19425 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
W20260929 19:02:13.016736 19295 tablet.cc:2369] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:02:13.118049 19164 tablet.cc:2369] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:02:13.142508 19426 tablet.cc:2369] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:13.249416 19446 raft_consensus.cc:482] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:13.249567 19446 raft_consensus.cc:504] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:13.250048 19446 leader_election.cc:288] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a3872008b954ee8aeebe43ec43393ed (127.0.98.2:41085), 90f3988879864ad593f1a70f1ac91c97 (127.0.98.3:44789)
I20260929 19:02:13.252446 19447 raft_consensus.cc:482] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:13.252561 19447 raft_consensus.cc:504] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:13.252902 19447 leader_election.cc:288] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a3872008b954ee8aeebe43ec43393ed (127.0.98.2:41085), a765888d8de24b829d7bf02ab268994a (127.0.98.1:38273)
I20260929 19:02:13.254308 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.250325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42147 (local address 127.0.98.3:44789)
0929 19:02:13.250367 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:13.250370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:13.250542 (+ 172us) server_negotiation.cc:407] Connection header received
0929 19:02:13.250583 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:13.250584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:13.250593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:13.250643 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:13.250858 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.251640 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.253322 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.253878 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.253927 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.253937 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:13.254201 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 19:02:13.254241 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:02:13.254308 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.250313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54667 (local address 127.0.98.2:41085)
0929 19:02:13.250367 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:13.250370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:13.250462 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:02:13.250537 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:13.250538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:13.250553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:13.250625 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:13.250894 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.251633 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.253321 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.253899 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.253936 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.253944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:13.254208 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 19:02:13.254243 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":23}
I20260929 19:02:13.254652 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.250266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44789 (local address 127.0.98.1:42147)
0929 19:02:13.250428 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:13.250437 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:13.250491 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:13.250660 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:13.250660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:13.250665 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:13.250773 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.250776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.251752 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.251752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.253199 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.253202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.254058 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.254059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.254117 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.254120 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:13.254430 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 19:02:13.254493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":115,"thread_start_us":58,"threads_started":1}
I20260929 19:02:13.254673 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.250229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41085 (local address 127.0.98.1:54667)
0929 19:02:13.250375 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:13.250387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:13.250469 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:13.250659 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:13.250660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:13.250662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:13.250773 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.250776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.251749 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.251750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.253199 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.253202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.254005 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.254007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.254107 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.254112 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:02:13.254430 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 19:02:13.254493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":94,"thread_start_us":59,"threads_started":1}
I20260929 19:02:13.255038 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "a765888d8de24b829d7bf02ab268994a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f3988879864ad593f1a70f1ac91c97" is_pre_election: true
I20260929 19:02:13.255096 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "a765888d8de24b829d7bf02ab268994a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3872008b954ee8aeebe43ec43393ed" is_pre_election: true
I20260929 19:02:13.255167 19379 raft_consensus.cc:2454] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a765888d8de24b829d7bf02ab268994a in term 0.
I20260929 19:02:13.255244 19248 raft_consensus.cc:2454] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a765888d8de24b829d7bf02ab268994a in term 0.
I20260929 19:02:13.255401 19053 leader_election.cc:302] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f3988879864ad593f1a70f1ac91c97, a765888d8de24b829d7bf02ab268994a; no voters:
I20260929 19:02:13.255515 19446 raft_consensus.cc:2781] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:13.255577 19446 raft_consensus.cc:482] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:13.255604 19446 raft_consensus.cc:3037] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:13.256551 19446 raft_consensus.cc:504] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:13.256706 19446 leader_election.cc:288] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [CANDIDATE]: Term 1 election: Requested vote from peers 7a3872008b954ee8aeebe43ec43393ed (127.0.98.2:41085), 90f3988879864ad593f1a70f1ac91c97 (127.0.98.3:44789)
I20260929 19:02:13.256743 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.253068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.98.3:35123)
0929 19:02:13.253351 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:13.253361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:13.253432 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:13.253606 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:13.253607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:13.253609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:13.253689 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.253690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.254573 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.254573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.255814 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.255816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.256504 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.256505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.256569 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.256573 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:13.256628 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:02:13.256663 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":20,"thread_start_us":87,"threads_started":1}
I20260929 19:02:13.256912 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "a765888d8de24b829d7bf02ab268994a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3872008b954ee8aeebe43ec43393ed"
I20260929 19:02:13.256994 19248 raft_consensus.cc:3037] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:13.257006 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "a765888d8de24b829d7bf02ab268994a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f3988879864ad593f1a70f1ac91c97"
I20260929 19:02:13.257076 19379 raft_consensus.cc:3037] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:13.257313 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.253024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41085 (local address 127.0.98.3:38005)
0929 19:02:13.253114 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:13.253125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:13.253163 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:13.253517 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:13.253518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:13.253531 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:13.253597 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.253599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.255471 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.255472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.256295 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:13.256297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.257103 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.257104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:13.257177 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.257180 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:13.257213 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 19:02:13.257255 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 19:02:13.257565 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.253294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38005 (local address 127.0.98.2:41085)
0929 19:02:13.253388 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:13.253390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:13.253396 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:02:13.253419 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:13.253420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:13.253430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:13.253663 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:13.253694 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.254783 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.256400 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.256988 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.257431 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.257442 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:13.257461 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 19:02:13.257500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260929 19:02:13.257921 19248 raft_consensus.cc:2454] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a765888d8de24b829d7bf02ab268994a in term 1.
I20260929 19:02:13.257964 19379 raft_consensus.cc:2454] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a765888d8de24b829d7bf02ab268994a in term 1.
I20260929 19:02:13.258160 19052 leader_election.cc:302] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a3872008b954ee8aeebe43ec43393ed, a765888d8de24b829d7bf02ab268994a; no voters:
I20260929 19:02:13.258275 19446 raft_consensus.cc:2781] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:13.258265 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "90f3988879864ad593f1a70f1ac91c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3872008b954ee8aeebe43ec43393ed" is_pre_election: true
I20260929 19:02:13.258358 19248 raft_consensus.cc:2379] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90f3988879864ad593f1a70f1ac91c97 in current term 1: Already voted for candidate a765888d8de24b829d7bf02ab268994a in this term.
I20260929 19:02:13.258498 19446 raft_consensus.cc:686] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [term 1 LEADER]: Becoming Leader. State: Replica: a765888d8de24b829d7bf02ab268994a, State: Running, Role: LEADER
I20260929 19:02:13.258569 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.253245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35123 (local address 127.0.98.1:38273)
0929 19:02:13.253283 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:13.253285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:13.253411 (+ 126us) server_negotiation.cc:407] Connection header received
0929 19:02:13.253506 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:13.253507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:13.253517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:13.253547 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:13.253771 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.254495 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.255906 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.256419 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.256496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.256506 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:13.256652 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 19:02:13.256686 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:13.258628 19446 consensus_queue.cc:237] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } }
I20260929 19:02:13.258816 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41eeac8021c4945ad0aea2094116f49" candidate_uuid: "90f3988879864ad593f1a70f1ac91c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a765888d8de24b829d7bf02ab268994a" is_pre_election: true
I20260929 19:02:13.259104 19315 leader_election.cc:302] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90f3988879864ad593f1a70f1ac91c97; no voters: 7a3872008b954ee8aeebe43ec43393ed, a765888d8de24b829d7bf02ab268994a
I20260929 19:02:13.259234 19447 raft_consensus.cc:2726] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 19:02:13.259552 18975 catalog_manager.cc:5495] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a reported cstate change: term changed from 0 to 1, leader changed from <none> to a765888d8de24b829d7bf02ab268994a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a765888d8de24b829d7bf02ab268994a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a765888d8de24b829d7bf02ab268994a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:13.272411 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.270012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.0.98.1:38273)
0929 19:02:13.270079 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:13.270081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:13.270089 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:13.270119 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:13.270120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:13.270136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:13.270194 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:13.270296 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.270767 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.271628 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:13.271847 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:13.271902 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:13.272091 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:13.272177 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:13.272250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:13.272297 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 19:02:13.272306 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:02:13.274227 19379 raft_consensus.cc:1261] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 1 FOLLOWER]: Refusing update from remote peer a765888d8de24b829d7bf02ab268994a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:13.274266 19248 raft_consensus.cc:1261] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 1 FOLLOWER]: Refusing update from remote peer a765888d8de24b829d7bf02ab268994a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:13.274550 19452 consensus_queue.cc:1035] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f3988879864ad593f1a70f1ac91c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:13.274680 19446 consensus_queue.cc:1035] T a41eeac8021c4945ad0aea2094116f49 P a765888d8de24b829d7bf02ab268994a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3872008b954ee8aeebe43ec43393ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:13.280079 18975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:13.280185 18975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:13.280905 18975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74b73709ba774b05939f39f161a3d159: Sending DeleteTablet for 3 replicas of tablet a41eeac8021c4945ad0aea2094116f49
I20260929 19:02:13.281219 19228 tablet_service.cc:1514] Processing DeleteTablet for tablet a41eeac8021c4945ad0aea2094116f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:13 UTC) from {username='slave'} at 127.0.0.1:37722
I20260929 19:02:13.281298 19359 tablet_service.cc:1514] Processing DeleteTablet for tablet a41eeac8021c4945ad0aea2094116f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:13 UTC) from {username='slave'} at 127.0.0.1:47748
I20260929 19:02:13.281531 19466 tablet_replica.cc:330] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: stopping tablet replica
I20260929 19:02:13.281606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19035
I20260929 19:02:13.281615 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet a41eeac8021c4945ad0aea2094116f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:13 UTC) from {username='slave'} at 127.0.0.1:38524
I20260929 19:02:13.282351 19465 tablet_replica.cc:330] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: stopping tablet replica
I20260929 19:02:13.282410 19466 raft_consensus.cc:2229] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:13.282433 19465 raft_consensus.cc:2229] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:13.282514 19466 raft_consensus.cc:2258] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:13.282513 19465 raft_consensus.cc:2258] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:13.282956 19466 ts_tablet_manager.cc:1905] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:13.282956 19465 ts_tablet_manager.cc:1905] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:13.284744 19465 ts_tablet_manager.cc:1918] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:13.284744 19466 ts_tablet_manager.cc:1918] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:13.284821 19466 log.cc:1196] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/a41eeac8021c4945ad0aea2094116f49
I20260929 19:02:13.284821 19465 log.cc:1196] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/a41eeac8021c4945ad0aea2094116f49
I20260929 19:02:13.285087 19466 ts_tablet_manager.cc:1939] T a41eeac8021c4945ad0aea2094116f49 P 90f3988879864ad593f1a70f1ac91c97: Deleting consensus metadata
I20260929 19:02:13.285087 19465 ts_tablet_manager.cc:1939] T a41eeac8021c4945ad0aea2094116f49 P 7a3872008b954ee8aeebe43ec43393ed: Deleting consensus metadata
I20260929 19:02:13.285584 18962 catalog_manager.cc:4841] TS 90f3988879864ad593f1a70f1ac91c97 (127.0.98.3:44789): tablet a41eeac8021c4945ad0aea2094116f49 (table test-table [id=e769d3d23cef4d468257bb999c19ccbb]) successfully deleted
I20260929 19:02:13.285590 18961 catalog_manager.cc:4841] TS 7a3872008b954ee8aeebe43ec43393ed (127.0.98.2:41085): tablet a41eeac8021c4945ad0aea2094116f49 (table test-table [id=e769d3d23cef4d468257bb999c19ccbb]) successfully deleted
W20260929 19:02:13.287166 18962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38273 (error 108)
I20260929 19:02:13.287456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19166
I20260929 19:02:13.287657 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:13.287540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.0.1:38532)
0929 19:02:13.287587 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:13.287621 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:02:13.288283 18962 catalog_manager.cc:4584] TS a765888d8de24b829d7bf02ab268994a (127.0.98.1:38273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a41eeac8021c4945ad0aea2094116f49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38273: connect: Connection refused (error 111)
I20260929 19:02:13.292670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19297
I20260929 19:02:13.297338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 18944
2026-09-29T19:02:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1009 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:13 dist-test-slave-gfsd krb5kdc[19470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:13 dist-test-slave-gfsd krb5kdc[19470](info): set up 2 sockets
Sep 29 19:02:13 dist-test-slave-gfsd krb5kdc[19470](info): commencing operation
krb5kdc: starting...
W20260929 19:02:14.343298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:14.461973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:14.544163 19483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:14.544394 19483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:14.544427 19483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:14.545531 19483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:14.545594 19483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:14.545619 19483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:14.545651 19483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:14.545684 19483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:14.547158 19483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:14.547446 19483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:14.547668 19483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:14.552474 19488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.552445 19491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.552479 19489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:14.552750 19483 server_base.cc:1034] running on GCE node
I20260929 19:02:14.552934 19483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:14.553158 19483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:14.554292 19483 hybrid_clock.cc:648] HybridClock initialized: now 1790708534554275 us; error 32 us; skew 500 ppm
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:14.556726 19483 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:14.557163 19483 webserver.cc:460] Webserver started at http://127.0.98.62:34869/ using document root <none> and password file <none>
I20260929 19:02:14.557339 19483 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:14.557379 19483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:14.557464 19483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:14.558411 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "3a35c04f303346e8bd6e001942699105"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "ee073cbdd796d5ebc9009810e432e6ef"
server_key_iv: "6e0e24b548fdb1fe46527d557b5273e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.558697 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a35c04f303346e8bd6e001942699105"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "ee073cbdd796d5ebc9009810e432e6ef"
server_key_iv: "6e0e24b548fdb1fe46527d557b5273e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.560014 19483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:14.560726 19498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.560945 19483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:14.561048 19483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "3a35c04f303346e8bd6e001942699105"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "ee073cbdd796d5ebc9009810e432e6ef"
server_key_iv: "6e0e24b548fdb1fe46527d557b5273e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.561127 19483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:14.583112 19483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:14.585021 19483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:14.585194 19483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:14.589290 19483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34637
I20260929 19:02:14.589329 19550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34637 every 8 connection(s)
I20260929 19:02:14.589699 19483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:14.590539 19551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:14.592917 19551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: Bootstrap starting.
I20260929 19:02:14.593493 19551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:14.593778 19551 log.cc:824] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:14.594478 19551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: No bootstrap required, opened a new log
I20260929 19:02:14.596187 19551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260929 19:02:14.596321 19551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:14.596344 19551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a35c04f303346e8bd6e001942699105, State: Initialized, Role: FOLLOWER
I20260929 19:02:14.596517 19551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260929 19:02:14.596577 19551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:14.596632 19551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:14.596674 19551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:14.596817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19483
I20260929 19:02:14.597014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:14.597222 392 external_mini_cluster.cc:1428] Setting key c42d1697fdbcffc1e32ab23ace18ccc5
I20260929 19:02:14.597330 19551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260929 19:02:14.597448 19551 leader_election.cc:302] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a35c04f303346e8bd6e001942699105; no voters:
I20260929 19:02:14.597613 19551 leader_election.cc:288] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:14.597653 19554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:14.597800 19554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [term 1 LEADER]: Becoming Leader. State: Replica: 3a35c04f303346e8bd6e001942699105, State: Running, Role: LEADER
I20260929 19:02:14.597875 19551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:14.597882 19554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260929 19:02:14.598400 19554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a35c04f303346e8bd6e001942699105. Latest consensus state: current_term: 1 leader_uuid: "3a35c04f303346e8bd6e001942699105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260929 19:02:14.598486 19554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:14.598623 19558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a35c04f303346e8bd6e001942699105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35c04f303346e8bd6e001942699105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260929 19:02:14.598701 19558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:14.598726 19561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:14.599244 19561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:14.600901 19561 catalog_manager.cc:1347] Generated new cluster ID: 467e19abd91e41af967030b19711d6ce
I20260929 19:02:14.600953 19561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:14.608987 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:14.597698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.0.98.62:34637)
0929 19:02:14.597997 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:14.598006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:14.598039 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:02:14.598224 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:14.598232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:14.598283 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:14.598461 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:14.599324 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.600734 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.601821 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.602474 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.602592 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:14.605373 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:14.605377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:14.605377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:14.605389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:14.607344 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.607739 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.607742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.607742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.607856 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.608073 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.608073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.608073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.608358 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:14.608509 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:14.608639 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 19:02:14.608761 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260929 19:02:14.620142 19561 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:14.620698 19561 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:14.626327 19561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: Generated new TSK 0
I20260929 19:02:14.626493 19561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:02:14.656102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:14.737310 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:14.737531 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:14.737571 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:14.738704 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:14.738772 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:14.738818 19577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:14.740402 19577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:14.740657 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:14.740895 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:14.745960 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.745975 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.746062 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:14.745990 19577 server_base.cc:1034] running on GCE node
I20260929 19:02:14.746414 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:14.746666 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:14.747839 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790708534747804 us; error 66 us; skew 500 ppm
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:14.750463 19577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:02:14.750869 19577 webserver.cc:460] Webserver started at http://127.0.98.1:38713/ using document root <none> and password file <none>
I20260929 19:02:14.751075 19577 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:14.751117 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:14.751205 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:14.752251 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "e449406425064104bf19e528e68f64bf"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "2b35873019b5e0c1f0d94116ba44b180"
server_key_iv: "2ce8b0321442ef0a611ff1c778a54c51"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.752543 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e449406425064104bf19e528e68f64bf"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "2b35873019b5e0c1f0d94116ba44b180"
server_key_iv: "2ce8b0321442ef0a611ff1c778a54c51"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.753770 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:02:14.754527 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.754700 19577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:14.754783 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "e449406425064104bf19e528e68f64bf"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "2b35873019b5e0c1f0d94116ba44b180"
server_key_iv: "2ce8b0321442ef0a611ff1c778a54c51"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.754838 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:14.779515 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:14.781644 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:14.781759 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:14.781955 19577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:14.782275 19577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:14.782310 19577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.782332 19577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:14.782380 19577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.788125 19577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37153
I20260929 19:02:14.788215 19705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37153 every 8 connection(s)
I20260929 19:02:14.788524 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:14.791054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19577
I20260929 19:02:14.791154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:14.791412 392 external_mini_cluster.cc:1428] Setting key 011fad1a339fcaebdaf36b3c906e9baa
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:14.799719 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:14.789380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58883 (local address 127.0.98.62:34637)
0929 19:02:14.789432 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:14.789434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:14.789676 (+ 242us) server_negotiation.cc:407] Connection header received
0929 19:02:14.790724 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:14.790725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:14.790742 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:14.790812 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:14.791664 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.792196 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.793303 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.793536 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.793592 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:14.796153 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:14.796154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:14.796155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:14.796163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:14.797972 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.798371 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.798372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.798372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.798442 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.798696 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.798697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.798697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.798852 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:14.798942 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:14.799497 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 19:02:14.799598 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:02:14.799757 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:14.789329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:58883)
0929 19:02:14.789542 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:14.789580 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:14.790598 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:14.790873 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:14.790877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:14.791227 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:14.791553 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:14.791559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.792320 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.792321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:14.793183 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:14.793187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.793626 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.793627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:14.793788 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:14.794328 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:14.794338 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:14.796053 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:14.798061 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:14.798067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:14.798069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:14.798286 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:14.798520 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:14.798521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:14.798521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:14.798623 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:14.799096 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:14.799103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:14.799231 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 19:02:14.799404 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 19:02:14.799590 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260929 19:02:14.800426 19706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260929 19:02:14.800532 19706 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:02:14.800765 19706 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
I20260929 19:02:14.801335 19515 ts_manager.cc:194] Registered new tserver with Master: e449406425064104bf19e528e68f64bf (127.0.98.1:37153)
I20260929 19:02:14.802105 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:02:14.817756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:14.899768 19711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:14.899996 19711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:14.900020 19711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:14.901120 19711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:14.901156 19711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:14.901182 19711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:14.902586 19711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:14.902758 19711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:14.902932 19711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:14.907896 19717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.907878 19716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:14.907933 19719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:14.908082 19711 server_base.cc:1034] running on GCE node
I20260929 19:02:14.908334 19711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:14.908531 19711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:14.909669 19711 hybrid_clock.cc:648] HybridClock initialized: now 1790708534909650 us; error 36 us; skew 500 ppm
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:14.912539 19711 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:02:14.912969 19711 webserver.cc:460] Webserver started at http://127.0.98.2:35197/ using document root <none> and password file <none>
I20260929 19:02:14.913199 19711 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:14.913268 19711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:14.913422 19711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:14.914513 19711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "899c91b7089f46c1ba46e292be89ca47"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "8dd180658dd9a9e06e612b232d38c6d2"
server_key_iv: "e1b49e15c4b6db7de2c6bed12e6a87ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.914880 19711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "899c91b7089f46c1ba46e292be89ca47"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "8dd180658dd9a9e06e612b232d38c6d2"
server_key_iv: "e1b49e15c4b6db7de2c6bed12e6a87ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.916332 19711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 19:02:14.917271 19726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.917474 19711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:14.917586 19711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "899c91b7089f46c1ba46e292be89ca47"
format_stamp: "Formatted at 2026-09-29 19:02:14 on dist-test-slave-gfsd"
server_key: "8dd180658dd9a9e06e612b232d38c6d2"
server_key_iv: "e1b49e15c4b6db7de2c6bed12e6a87ea"
server_key_version: "encryptionkey@0"
I20260929 19:02:14.917661 19711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:14.931475 19711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:14.933800 19711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:14.933974 19711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:14.934206 19711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:14.934516 19711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:14.934571 19711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.934609 19711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:14.934638 19711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:14.940392 19711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42783
I20260929 19:02:14.940443 19839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42783 every 8 connection(s)
I20260929 19:02:14.940737 19711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:14.942399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19711
I20260929 19:02:14.942540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:14.942716 392 external_mini_cluster.cc:1428] Setting key a7fbaa4fa7f383ca444b01090712ecf8
Sep 29 19:02:14 dist-test-slave-gfsd krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 19:02:14.950580 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:14.941446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34697 (local address 127.0.98.62:34637)
0929 19:02:14.941511 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:14.941513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:14.941708 (+ 195us) server_negotiation.cc:407] Connection header received
0929 19:02:14.942817 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:14.942818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:14.942830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:14.942889 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:14.943700 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.944206 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.945215 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.945430 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.945516 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:14.947697 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:14.947699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:14.947699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:14.947707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:14.949186 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.949590 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.949591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.949592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.949656 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:14.949908 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:14.949909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:14.949909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:14.950033 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:14.950111 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:14.950380 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 19:02:14.950477 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:02:14.950758 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:14.941439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:34697)
0929 19:02:14.941590 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:14.941625 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:14.942709 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:14.943060 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:14.943063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:14.943370 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:14.943606 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:14.943612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.944303 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.944304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:14.945112 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:14.945114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:14.945524 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:14.945525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:14.945651 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:14.946187 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:14.946195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:14.947599 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:14.949279 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:14.949286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:14.949289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:14.949511 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:14.949724 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:14.949725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:14.949725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:14.949838 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:14.950101 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:14.950108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:14.950251 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 19:02:14.950359 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:02:14.950515 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260929 19:02:14.951254 19840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260929 19:02:14.951391 19840 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:14.951599 19840 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
I20260929 19:02:14.952011 19515 ts_manager.cc:194] Registered new tserver with Master: 899c91b7089f46c1ba46e292be89ca47 (127.0.98.2:42783)
I20260929 19:02:14.952345 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:02:14.967993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:15.052793 19845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:15.053002 19845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:15.053027 19845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:15.054144 19845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:15.054188 19845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:15.054216 19845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:15.055783 19845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:15.055990 19845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:15.056180 19845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:15.061442 19851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.061525 19850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.061686 19853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.061671 19845 server_base.cc:1034] running on GCE node
I20260929 19:02:15.062026 19845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:15.062305 19845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:15.063505 19845 hybrid_clock.cc:648] HybridClock initialized: now 1790708535063494 us; error 28 us; skew 500 ppm
Sep 29 19:02:15 dist-test-slave-gfsd krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:15.066082 19845 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:02:15.066578 19845 webserver.cc:460] Webserver started at http://127.0.98.3:43705/ using document root <none> and password file <none>
I20260929 19:02:15.066819 19845 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:15.066895 19845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:15.067061 19845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:15.068188 19845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "9dc938b1a1564a349678560f1292886c"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "19cf26a18b376fedcfc829e7d847a232"
server_key_iv: "2307dbfd4bf603c92ea52c142fb08911"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.068548 19845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9dc938b1a1564a349678560f1292886c"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "19cf26a18b376fedcfc829e7d847a232"
server_key_iv: "2307dbfd4bf603c92ea52c142fb08911"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.069952 19845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:02:15.070777 19860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.071034 19845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:15.071142 19845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "9dc938b1a1564a349678560f1292886c"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "19cf26a18b376fedcfc829e7d847a232"
server_key_iv: "2307dbfd4bf603c92ea52c142fb08911"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.071228 19845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:15.085312 19845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:15.087723 19845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:15.087894 19845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:15.088143 19845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:15.088459 19845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:15.088519 19845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.088562 19845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:15.088593 19845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.094461 19845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46647
I20260929 19:02:15.094523 19973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46647 every 8 connection(s)
I20260929 19:02:15.094875 19845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:02:15 dist-test-slave-gfsd krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:15.102921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 19845
I20260929 19:02:15.103041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:15.103235 392 external_mini_cluster.cc:1428] Setting key 33e50c8ba11d45c7e5e203cdf26d8818
I20260929 19:02:15.104957 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.095670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54869 (local address 127.0.98.62:34637)
0929 19:02:15.095782 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.095785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.096036 (+ 251us) server_negotiation.cc:407] Connection header received
0929 19:02:15.097039 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.097041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.097053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.097100 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.097743 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.098330 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.099506 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.099699 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.099783 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.101932 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.101934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.101934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:15.101940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.103462 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:15.103931 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:15.103933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:15.103934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:15.104007 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:15.104278 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:15.104279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:15.104280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:15.104438 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.104504 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.104768 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.104864 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 19:02:15.105134 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.095647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:54869)
0929 19:02:15.095907 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.095944 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.096915 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.097181 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.097185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.097448 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:15.097635 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.097640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.098423 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.098425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.099410 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.099417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.099791 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.099793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.099919 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.100464 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:15.100473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:15.101830 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:15.103537 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:15.103544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:15.103546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:15.103847 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:15.104080 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:15.104081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:15.104082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:15.104216 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:15.104495 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:15.104501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:15.104638 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.104748 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.104900 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260929 19:02:15.105625 19974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260929 19:02:15.105703 19974 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:15.105909 19974 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
I20260929 19:02:15.106271 19515 ts_manager.cc:194] Registered new tserver with Master: 9dc938b1a1564a349678560f1292886c (127.0.98.3:46647)
I20260929 19:02:15.106649 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54869
I20260929 19:02:15.115314 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:15.122371 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.116740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.0.98.62:34637)
0929 19:02:15.116803 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.116805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.116859 (+ 54us) server_negotiation.cc:407] Connection header received
0929 19:02:15.116929 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.116930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.116937 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.116992 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.117501 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.117976 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.118680 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.118854 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.118916 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.120124 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.120125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.120125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:15.120131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.121468 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:15.121758 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:15.121759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:15.121759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:15.121805 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:15.121979 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:15.121979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:15.121979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:15.122105 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.122178 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.122261 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.122313 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:02:15.124060 19515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:15.125602 19515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:15.131803 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.127839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46647 (local address 127.0.0.1:48252)
0929 19:02:15.128520 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.128532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.128584 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.128847 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.128848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.128850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.128903 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.128904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129757 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.129758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.130749 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.130752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.131518 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.131519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.131595 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.131609 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.131684 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.131730 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260929 19:02:15.131827 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.128030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.0.98.3:46647)
0929 19:02:15.128193 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.128198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.128644 (+ 446us) server_negotiation.cc:407] Connection header received
0929 19:02:15.128684 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.128686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.128700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.128768 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.128989 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129654 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.130826 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.131436 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.131509 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.131563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.131710 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.131747 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260929 19:02:15.132292 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.127818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37153 (local address 127.0.0.1:36234)
0929 19:02:15.128147 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.128159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.128206 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.128527 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.128528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.128529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.128573 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.128574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129869 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.129870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.131079 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.131082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.132041 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.132042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.132107 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.132111 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.132164 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.132202 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":46,"thread_start_us":49,"threads_started":1}
I20260929 19:02:15.132380 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.127839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42783 (local address 127.0.0.1:40228)
0929 19:02:15.128095 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.128136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.128206 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.128440 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.128443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.128446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.128510 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.128512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129829 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.129830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.131079 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.131082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.132160 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.132160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.132228 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.132231 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.132281 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.132319 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":43,"thread_start_us":112,"threads_started":1}
I20260929 19:02:15.132567 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.127928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.98.1:37153)
0929 19:02:15.128346 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.128349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.128357 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:15.128411 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.128414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.128430 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.128645 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.128724 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129752 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.131177 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.131969 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.132039 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.132092 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.132428 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.132481 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":79,"threads_started":1}
I20260929 19:02:15.132890 19908 tablet_service.cc:1467] Processing CreateTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c (DEFAULT_TABLE table=test-table [id=afb3d1902ae048218c0af1efdab805af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.133241 19908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8c4de0fb55424eb0bc497a2f1df23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.133214 19640 tablet_service.cc:1467] Processing CreateTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c (DEFAULT_TABLE table=test-table [id=afb3d1902ae048218c0af1efdab805af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.133529 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8c4de0fb55424eb0bc497a2f1df23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.135425 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.128073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.2:42783)
0929 19:02:15.128290 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.128292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.128300 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:15.128332 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.128333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.128346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.128424 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.128598 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.129696 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.131354 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.132090 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.132161 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.132220 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.132297 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.132341 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260929 19:02:15.136058 19774 tablet_service.cc:1467] Processing CreateTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c (DEFAULT_TABLE table=test-table [id=afb3d1902ae048218c0af1efdab805af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.136405 19774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8c4de0fb55424eb0bc497a2f1df23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.136670 19988 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: Bootstrap starting.
I20260929 19:02:15.137240 19988 tablet_bootstrap.cc:654] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.137719 19988 log.cc:824] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.138049 19989 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Bootstrap starting.
I20260929 19:02:15.138440 19988 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: No bootstrap required, opened a new log
I20260929 19:02:15.138510 19988 ts_tablet_manager.cc:1397] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:15.138677 19989 tablet_bootstrap.cc:654] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.139205 19989 log.cc:824] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.139767 19991 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Bootstrap starting.
I20260929 19:02:15.139964 19989 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: No bootstrap required, opened a new log
I20260929 19:02:15.140043 19989 ts_tablet_manager.cc:1397] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:02:15.140352 19991 tablet_bootstrap.cc:654] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.140918 19991 log.cc:824] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.140854 19988 raft_consensus.cc:348] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.141100 19988 raft_consensus.cc:374] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.141152 19988 raft_consensus.cc:729] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e449406425064104bf19e528e68f64bf, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.141264 19988 consensus_queue.cc:260] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.141600 19706 heartbeater.cc:503] Master 127.0.98.62:34637 was elected leader, sending a full tablet report...
I20260929 19:02:15.141583 19988 ts_tablet_manager.cc:1428] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:15.141803 19989 raft_consensus.cc:348] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.141999 19989 raft_consensus.cc:374] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.142047 19989 raft_consensus.cc:729] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc938b1a1564a349678560f1292886c, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.142155 19989 consensus_queue.cc:260] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.142413 19989 ts_tablet_manager.cc:1428] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:02:15.142438 19974 heartbeater.cc:503] Master 127.0.98.62:34637 was elected leader, sending a full tablet report...
I20260929 19:02:15.143008 19991 tablet_bootstrap.cc:492] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: No bootstrap required, opened a new log
I20260929 19:02:15.143095 19991 ts_tablet_manager.cc:1397] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:02:15.144573 19991 raft_consensus.cc:348] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.144727 19991 raft_consensus.cc:374] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.144771 19991 raft_consensus.cc:729] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899c91b7089f46c1ba46e292be89ca47, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.144874 19991 consensus_queue.cc:260] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.145123 19991 ts_tablet_manager.cc:1428] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:15.145151 19840 heartbeater.cc:503] Master 127.0.98.62:34637 was elected leader, sending a full tablet report...
W20260929 19:02:15.191303 19841 tablet.cc:2369] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:15.197724 19995 raft_consensus.cc:482] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:15.197867 19995 raft_consensus.cc:504] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.198256 19995 leader_election.cc:288] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e449406425064104bf19e528e68f64bf (127.0.98.1:37153), 899c91b7089f46c1ba46e292be89ca47 (127.0.98.2:42783)
I20260929 19:02:15.202327 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.198577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42783 (local address 127.0.98.3:54793)
0929 19:02:15.198660 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.198674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.198723 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.198903 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.198904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.198906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.198983 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.198985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.199963 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.199965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.201286 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.201288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.202114 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.202115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.202188 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.202192 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.202240 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.202267 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":2}
I20260929 19:02:15.202353 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.198700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54793 (local address 127.0.98.2:42783)
0929 19:02:15.198753 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.198755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.198765 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:15.198794 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.198795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.198806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.198898 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.199060 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.199879 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.201393 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.202039 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.202102 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.202110 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.202258 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.202294 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:02:15.202335 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.198619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37153 (local address 127.0.98.3:45427)
0929 19:02:15.198800 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.198809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.198948 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.199120 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.199120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.199122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.199170 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.199171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.200023 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.200024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.201286 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.201288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.202039 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.202040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.202125 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.202129 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.202188 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.202237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260929 19:02:15.202333 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.198701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45427 (local address 127.0.98.1:37153)
0929 19:02:15.198753 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.198755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.198935 (+ 180us) server_negotiation.cc:407] Connection header received
0929 19:02:15.199031 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.199032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.199046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.199097 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.199237 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.199953 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.201395 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.201952 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.202032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.202044 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.202205 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.202267 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:02:15.202684 19794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8c4de0fb55424eb0bc497a2f1df23c" candidate_uuid: "9dc938b1a1564a349678560f1292886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899c91b7089f46c1ba46e292be89ca47" is_pre_election: true
I20260929 19:02:15.202726 19660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8c4de0fb55424eb0bc497a2f1df23c" candidate_uuid: "9dc938b1a1564a349678560f1292886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e449406425064104bf19e528e68f64bf" is_pre_election: true
I20260929 19:02:15.202854 19660 raft_consensus.cc:2454] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc938b1a1564a349678560f1292886c in term 0.
I20260929 19:02:15.202854 19794 raft_consensus.cc:2454] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc938b1a1564a349678560f1292886c in term 0.
I20260929 19:02:15.203132 19862 leader_election.cc:302] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc938b1a1564a349678560f1292886c, e449406425064104bf19e528e68f64bf; no voters:
I20260929 19:02:15.203297 19995 raft_consensus.cc:2781] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:15.203348 19995 raft_consensus.cc:482] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:15.203372 19995 raft_consensus.cc:3037] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.204098 19995 raft_consensus.cc:504] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.204248 19995 leader_election.cc:288] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [CANDIDATE]: Term 1 election: Requested vote from peers e449406425064104bf19e528e68f64bf (127.0.98.1:37153), 899c91b7089f46c1ba46e292be89ca47 (127.0.98.2:42783)
I20260929 19:02:15.204383 19660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8c4de0fb55424eb0bc497a2f1df23c" candidate_uuid: "9dc938b1a1564a349678560f1292886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e449406425064104bf19e528e68f64bf"
I20260929 19:02:15.204459 19660 raft_consensus.cc:3037] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.204449 19794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8c4de0fb55424eb0bc497a2f1df23c" candidate_uuid: "9dc938b1a1564a349678560f1292886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899c91b7089f46c1ba46e292be89ca47"
I20260929 19:02:15.204531 19794 raft_consensus.cc:3037] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.205165 19660 raft_consensus.cc:2454] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc938b1a1564a349678560f1292886c in term 1.
I20260929 19:02:15.205336 19794 raft_consensus.cc:2454] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc938b1a1564a349678560f1292886c in term 1.
I20260929 19:02:15.205350 19862 leader_election.cc:302] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc938b1a1564a349678560f1292886c, e449406425064104bf19e528e68f64bf; no voters:
I20260929 19:02:15.205454 19995 raft_consensus.cc:2781] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:15.205595 19995 raft_consensus.cc:686] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 1 LEADER]: Becoming Leader. State: Replica: 9dc938b1a1564a349678560f1292886c, State: Running, Role: LEADER
I20260929 19:02:15.205699 19995 consensus_queue.cc:237] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } }
I20260929 19:02:15.206643 19514 catalog_manager.cc:5495] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dc938b1a1564a349678560f1292886c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9dc938b1a1564a349678560f1292886c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc938b1a1564a349678560f1292886c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:15.224500 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.221968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.0.98.3:46647)
0929 19:02:15.222028 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.222030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.222042 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:02:15.222093 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.222094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.222112 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.222191 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:15.222332 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.222805 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.223699 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.223898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.223975 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.224150 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:15.224253 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:15.224352 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.224368 (+ 16us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.224391 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:02:15.226521 19794 raft_consensus.cc:1261] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 1 FOLLOWER]: Refusing update from remote peer 9dc938b1a1564a349678560f1292886c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:15.226521 19660 raft_consensus.cc:1261] T cd8c4de0fb55424eb0bc497a2f1df23c P e449406425064104bf19e528e68f64bf [term 1 FOLLOWER]: Refusing update from remote peer 9dc938b1a1564a349678560f1292886c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:15.226866 19995 consensus_queue.cc:1035] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e449406425064104bf19e528e68f64bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:15.227007 19998 consensus_queue.cc:1035] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [LEADER]: Connected to new peer: Peer: permanent_uuid: "899c91b7089f46c1ba46e292be89ca47" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:15.229449 20002 mvcc.cc:205] Tried to move back new op lower bound from 7334742160284377088 to 7334742160203530240. Current Snapshot: MvccSnapshot[applied={T|T < 7334742160284377088}]
I20260929 19:02:15.230690 20003 mvcc.cc:205] Tried to move back new op lower bound from 7334742160284377088 to 7334742160203530240. Current Snapshot: MvccSnapshot[applied={T|T < 7334742160284377088 or (T in {7334742160284377088})}]
I20260929 19:02:15.232136 19515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:15.232232 19515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:15.233052 19515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a35c04f303346e8bd6e001942699105: Sending DeleteTablet for 3 replicas of tablet cd8c4de0fb55424eb0bc497a2f1df23c
I20260929 19:02:15.233378 19908 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48252
I20260929 19:02:15.233413 19640 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36234
I20260929 19:02:15.233608 19774 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8c4de0fb55424eb0bc497a2f1df23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40228
I20260929 19:02:15.233814 20013 tablet_replica.cc:330] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: stopping tablet replica
I20260929 19:02:15.233827 20011 tablet_replica.cc:330] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: stopping tablet replica
I20260929 19:02:15.233880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19577
I20260929 19:02:15.233922 20011 raft_consensus.cc:2229] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:15.234035 20011 raft_consensus.cc:2258] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:15.234045 20013 raft_consensus.cc:2229] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:15.234238 20013 raft_consensus.cc:2258] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:15.234445 20011 ts_tablet_manager.cc:1905] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:15.236444 20011 ts_tablet_manager.cc:1918] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:15.236522 20011 log.cc:1196] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/cd8c4de0fb55424eb0bc497a2f1df23c
I20260929 19:02:15.236641 20013 ts_tablet_manager.cc:1905] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:15.236788 20011 ts_tablet_manager.cc:1939] T cd8c4de0fb55424eb0bc497a2f1df23c P 9dc938b1a1564a349678560f1292886c: Deleting consensus metadata
I20260929 19:02:15.237442 19501 catalog_manager.cc:4841] TS 9dc938b1a1564a349678560f1292886c (127.0.98.3:46647): tablet cd8c4de0fb55424eb0bc497a2f1df23c (table test-table [id=afb3d1902ae048218c0af1efdab805af]) successfully deleted
I20260929 19:02:15.238360 20013 ts_tablet_manager.cc:1918] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:15.238451 20013 log.cc:1196] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708293967505-392-0/minicluster-data/ts-1/wal/wals/cd8c4de0fb55424eb0bc497a2f1df23c
I20260929 19:02:15.238706 20013 ts_tablet_manager.cc:1939] T cd8c4de0fb55424eb0bc497a2f1df23c P 899c91b7089f46c1ba46e292be89ca47: Deleting consensus metadata
I20260929 19:02:15.239149 19502 catalog_manager.cc:4841] TS 899c91b7089f46c1ba46e292be89ca47 (127.0.98.2:42783): tablet cd8c4de0fb55424eb0bc497a2f1df23c (table test-table [id=afb3d1902ae048218c0af1efdab805af]) successfully deleted
W20260929 19:02:15.243054 19500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37153 (error 108)
I20260929 19:02:15.243317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19711
I20260929 19:02:15.243995 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.243474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37153 (local address 127.0.0.1:36246)
0929 19:02:15.243908 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.243954 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409}
W20260929 19:02:15.244591 19500 catalog_manager.cc:4584] TS e449406425064104bf19e528e68f64bf (127.0.98.1:37153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd8c4de0fb55424eb0bc497a2f1df23c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37153: connect: Connection refused (error 111)
I20260929 19:02:15.248529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19845
I20260929 19:02:15.252735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 19483
2026-09-29T19:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1955 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T19:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:15Z Disabled control of system clock
I20260929 19:02:15.280241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39823
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:15.360792 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:15.361022 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:15.361043 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:15.362116 20018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:15.362152 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:15.362165 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:15.362177 20018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:15.362190 20018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:15.363483 20018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:15.363679 20018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:15.363999 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:15.368932 20024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.368974 20026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.369006 20018 server_base.cc:1034] running on GCE node
W20260929 19:02:15.368976 20023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.369442 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:15.369699 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:15.370875 20018 hybrid_clock.cc:648] HybridClock initialized: now 1790708535370855 us; error 34 us; skew 500 ppm
I20260929 19:02:15.372303 20018 webserver.cc:460] Webserver started at http://127.0.98.62:37817/ using document root <none> and password file <none>
I20260929 19:02:15.372571 20018 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:15.372650 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:15.372787 20018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:15.373849 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "3ceb89194bbf4960b83ab85286e71ccd"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "bc7bfe04d8c394bcda47acedfd6c3b7f"
server_key_iv: "2ec2e96bcdc8272d814ef9b2f73e1428"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.374243 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ceb89194bbf4960b83ab85286e71ccd"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "bc7bfe04d8c394bcda47acedfd6c3b7f"
server_key_iv: "2ec2e96bcdc8272d814ef9b2f73e1428"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.375602 20018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:15.376370 20032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.376554 20018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:15.376652 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "3ceb89194bbf4960b83ab85286e71ccd"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "bc7bfe04d8c394bcda47acedfd6c3b7f"
server_key_iv: "2ec2e96bcdc8272d814ef9b2f73e1428"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.376721 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:15.401414 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:15.401767 20018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:15.401942 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:15.406311 20018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39823
I20260929 19:02:15.406353 20084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39823 every 8 connection(s)
I20260929 19:02:15.406750 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:15.407785 20085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.410066 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: Bootstrap starting.
I20260929 19:02:15.410665 20085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.410974 20085 log.cc:824] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.411762 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: No bootstrap required, opened a new log
I20260929 19:02:15.413432 20085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } }
I20260929 19:02:15.413569 20085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.413592 20085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ceb89194bbf4960b83ab85286e71ccd, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.413745 20085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } }
I20260929 19:02:15.413817 20085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:15.413838 20085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:15.413914 20085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.414429 20085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } }
I20260929 19:02:15.414547 20085 leader_election.cc:302] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ceb89194bbf4960b83ab85286e71ccd; no voters:
I20260929 19:02:15.414748 20085 leader_election.cc:288] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:15.414783 20088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:15.414801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20018
I20260929 19:02:15.414901 20088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [term 1 LEADER]: Becoming Leader. State: Replica: 3ceb89194bbf4960b83ab85286e71ccd, State: Running, Role: LEADER
I20260929 19:02:15.414913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:15.414989 20088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } }
I20260929 19:02:15.415117 392 external_mini_cluster.cc:1428] Setting key 9651d42ef2e9be96f06d86c7d7461155
I20260929 19:02:15.415025 20085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:15.415360 20089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ceb89194bbf4960b83ab85286e71ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } } }
I20260929 19:02:15.415521 20089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:15.415787 20090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ceb89194bbf4960b83ab85286e71ccd. Latest consensus state: current_term: 1 leader_uuid: "3ceb89194bbf4960b83ab85286e71ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb89194bbf4960b83ab85286e71ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39823 } } }
I20260929 19:02:15.415884 20090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:15.416112 20093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:15.416702 20093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:15.418259 20093 catalog_manager.cc:1347] Generated new cluster ID: 2be6e6b14c3f45e689f1b0dec61fdc9c
I20260929 19:02:15.418316 20093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:15.421226 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.415752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.0.98.62:39823)
0929 19:02:15.415981 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.415993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.416032 (+ 39us) server_negotiation.cc:407] Connection header received
0929 19:02:15.416150 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.416157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.416233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.416839 (+ 606us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.416952 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.418283 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.419186 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.419699 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.419839 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.420574 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.420578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.420578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:15.420755 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.420787 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:15.420798 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.420877 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.420943 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.421022 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260929 19:02:15.439217 20093 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:15.439774 20093 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:15.447853 20093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: Generated new TSK 0
I20260929 19:02:15.448042 20093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:15.454734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--builtin_ntp_servers=127.0.98.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:15.542837 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:15.543044 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:15.543066 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:15.544299 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:15.544341 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:15.544366 20109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:15.545709 20109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:15.545961 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:15.546276 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:15.552443 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.552462 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.552461 20109 server_base.cc:1034] running on GCE node
W20260929 19:02:15.552433 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.552891 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:15.553210 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:15.554385 20109 hybrid_clock.cc:648] HybridClock initialized: now 1790708535554361 us; error 39 us; skew 500 ppm
I20260929 19:02:15.555851 20109 webserver.cc:460] Webserver started at http://127.0.98.1:41959/ using document root <none> and password file <none>
I20260929 19:02:15.556124 20109 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:15.556203 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:15.556355 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:15.557456 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data/instance:
uuid: "59503aafcd4142fd9d945c1102c2a0b5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "719a5048d7362a31f01b1624b6f48562"
server_key_iv: "a43c770d271f39f860067e314da8891c"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.557807 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59503aafcd4142fd9d945c1102c2a0b5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "719a5048d7362a31f01b1624b6f48562"
server_key_iv: "a43c770d271f39f860067e314da8891c"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.559185 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:15.559952 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.560132 20109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:02:15.560237 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal
uuid: "59503aafcd4142fd9d945c1102c2a0b5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "719a5048d7362a31f01b1624b6f48562"
server_key_iv: "a43c770d271f39f860067e314da8891c"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.560323 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:15.572763 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:15.573055 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:15.573203 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:15.573422 20109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:15.573728 20109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:15.573781 20109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.573822 20109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:15.573849 20109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.579782 20109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32865
I20260929 19:02:15.579833 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32865 every 8 connection(s)
I20260929 19:02:15.580211 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:15.585521 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.581005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38847 (local address 127.0.98.62:39823)
0929 19:02:15.581077 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.581079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.581387 (+ 308us) server_negotiation.cc:407] Connection header received
0929 19:02:15.581880 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.581881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.581893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.581950 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.582582 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.583137 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.584187 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.584389 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.584457 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.585269 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.585270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.585271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:15.585274 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.585287 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:15.585291 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.585337 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.585433 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.585461 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:02:15.585714 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.581088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39823 (local address 127.0.98.1:38847)
0929 19:02:15.581248 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.581307 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.581791 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.582009 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.582013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.582021 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:15.582508 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.582512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.583216 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.583217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.584088 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.584091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.584493 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.584494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.584611 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.585121 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:15.585122 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:15.585150 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:15.585171 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:15.585350 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:15.585360 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:15.585363 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.585427 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.585507 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260929 19:02:15.586279 20237 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39823
I20260929 19:02:15.586392 20237 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:15.586544 20237 heartbeater.cc:511] Master 127.0.98.62:39823 requested a full tablet report, sending...
I20260929 19:02:15.587052 20049 ts_manager.cc:194] Registered new tserver with Master: 59503aafcd4142fd9d945c1102c2a0b5 (127.0.98.1:32865)
I20260929 19:02:15.587719 20049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38847
I20260929 19:02:15.590265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20109
I20260929 19:02:15.590372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:15.590626 392 external_mini_cluster.cc:1428] Setting key 5bb07a62fd1c001bda313c0e9cdeaf48
I20260929 19:02:15.591555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--builtin_ntp_servers=127.0.98.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:02:15.671588 20240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:15.671803 20240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:15.671839 20240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:15.672968 20240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:15.673029 20240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:15.673071 20240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:15.674401 20240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:15.674651 20240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:15.674934 20240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:15.680007 20246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.680058 20245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.680178 20248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.680315 20240 server_base.cc:1034] running on GCE node
I20260929 19:02:15.680554 20240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:15.680836 20240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:15.682016 20240 hybrid_clock.cc:648] HybridClock initialized: now 1790708535682029 us; error 69 us; skew 500 ppm
I20260929 19:02:15.683359 20240 webserver.cc:460] Webserver started at http://127.0.98.2:39157/ using document root <none> and password file <none>
I20260929 19:02:15.683607 20240 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:15.683676 20240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:15.683809 20240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:15.684921 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data/instance:
uuid: "7564f84710d743f28f37aa9c38578e9d"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "6d4e243a8d51d978bec4e381a836bdcf"
server_key_iv: "90d27df7060f85a6fe9b3bcf17b3180a"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.685281 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7564f84710d743f28f37aa9c38578e9d"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "6d4e243a8d51d978bec4e381a836bdcf"
server_key_iv: "90d27df7060f85a6fe9b3bcf17b3180a"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.686570 20240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:15.687500 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.687671 20240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:15.687774 20240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal
uuid: "7564f84710d743f28f37aa9c38578e9d"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "6d4e243a8d51d978bec4e381a836bdcf"
server_key_iv: "90d27df7060f85a6fe9b3bcf17b3180a"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.687866 20240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:15.696903 20240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:15.697190 20240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:15.697340 20240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:15.697566 20240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:15.697883 20240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:15.697937 20240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.697978 20240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:15.698004 20240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.703603 20240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41023
I20260929 19:02:15.703670 20367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41023 every 8 connection(s)
I20260929 19:02:15.704001 20240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:15.706236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20240
I20260929 19:02:15.706352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:15.706571 392 external_mini_cluster.cc:1428] Setting key 47640e10a77bf35294eec9ab821c97e5
I20260929 19:02:15.707568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--builtin_ntp_servers=127.0.98.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:02:15.709810 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.704806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38211 (local address 127.0.98.62:39823)
0929 19:02:15.704924 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.704927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.705084 (+ 157us) server_negotiation.cc:407] Connection header received
0929 19:02:15.705552 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.705553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.705562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.705623 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.706253 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.707012 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.708202 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.708528 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.708606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.709579 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.709580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.709581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:15.709585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.709600 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:15.709603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.709643 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.709729 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.709753 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 19:02:15.710105 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.704743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39823 (local address 127.0.98.2:38211)
0929 19:02:15.704939 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.704980 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.705469 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.705673 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.705676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.705685 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:15.706167 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.706171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.707105 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.707106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.708122 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.708124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.708615 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.708616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.708751 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.709419 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:15.709420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:15.709460 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:15.709490 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:15.709654 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:15.709659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:15.709662 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.709727 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.709840 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260929 19:02:15.710628 20368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39823
I20260929 19:02:15.710719 20368 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:15.710911 20368 heartbeater.cc:511] Master 127.0.98.62:39823 requested a full tablet report, sending...
I20260929 19:02:15.711319 20049 ts_manager.cc:194] Registered new tserver with Master: 7564f84710d743f28f37aa9c38578e9d (127.0.98.2:41023)
I20260929 19:02:15.711655 20049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38211
W20260929 19:02:15.789424 20371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:15.789669 20371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:15.789703 20371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:15.790964 20371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:15.791033 20371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:15.791076 20371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:15.792680 20371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:15.792977 20371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:15.793306 20371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:15.798761 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.798746 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:15.798746 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:15.798928 20371 server_base.cc:1034] running on GCE node
I20260929 19:02:15.799314 20371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:15.799582 20371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:15.800822 20371 hybrid_clock.cc:648] HybridClock initialized: now 1790708535800775 us; error 64 us; skew 500 ppm
I20260929 19:02:15.802160 20371 webserver.cc:460] Webserver started at http://127.0.98.3:41709/ using document root <none> and password file <none>
I20260929 19:02:15.802484 20371 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:15.802549 20371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:15.802695 20371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:15.803889 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data/instance:
uuid: "1580f5150b88441f8d2b835c79a7d9f5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "ca41f44e1bf8dc172c9100bb84d358ce"
server_key_iv: "a7e95ca4816a52637258ef04d48211c5"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.804262 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1580f5150b88441f8d2b835c79a7d9f5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "ca41f44e1bf8dc172c9100bb84d358ce"
server_key_iv: "a7e95ca4816a52637258ef04d48211c5"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.805613 20371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:15.806417 20385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.806631 20371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:15.806721 20371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal
uuid: "1580f5150b88441f8d2b835c79a7d9f5"
format_stamp: "Formatted at 2026-09-29 19:02:15 on dist-test-slave-gfsd"
server_key: "ca41f44e1bf8dc172c9100bb84d358ce"
server_key_iv: "a7e95ca4816a52637258ef04d48211c5"
server_key_version: "encryptionkey@0"
I20260929 19:02:15.806797 20371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:15.818655 20371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:15.818953 20371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:15.819108 20371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:15.819380 20371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:15.819716 20371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:15.819749 20371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.819800 20371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:15.819828 20371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:15.825562 20371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36201
I20260929 19:02:15.825635 20498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36201 every 8 connection(s)
I20260929 19:02:15.825963 20371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:15.831298 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.826687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55141 (local address 127.0.98.62:39823)
0929 19:02:15.826729 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.826732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.826981 (+ 249us) server_negotiation.cc:407] Connection header received
0929 19:02:15.827578 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.827580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.827592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.827647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.828315 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.828822 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.829804 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.830009 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.830089 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.830978 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.830980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.830980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:15.830986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.831006 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:15.831010 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.831071 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.831166 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.831198 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:15.831516 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.826667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39823 (local address 127.0.98.3:55141)
0929 19:02:15.826854 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.826887 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.827471 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.827716 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.827720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.827730 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:15.828222 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.828226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.828926 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.828927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.829697 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.829699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.830095 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.830096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.830230 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.830803 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:02:15.830804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:15.830833 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:02:15.830853 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:15.831072 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:15.831079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:15.831082 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.831170 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.831268 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260929 19:02:15.832089 20499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39823
I20260929 19:02:15.832185 20499 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:15.832322 20499 heartbeater.cc:511] Master 127.0.98.62:39823 requested a full tablet report, sending...
I20260929 19:02:15.832675 20049 ts_manager.cc:194] Registered new tserver with Master: 1580f5150b88441f8d2b835c79a7d9f5 (127.0.98.3:36201)
I20260929 19:02:15.832984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20371
I20260929 19:02:15.833086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:15.833173 20049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55141
I20260929 19:02:15.833282 392 external_mini_cluster.cc:1428] Setting key e06bde6431d2f63d06bb2a91aef972e4
I20260929 19:02:15.834435 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:15.838599 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.835812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.62:39823)
0929 19:02:15.835881 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.835883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.835961 (+ 78us) server_negotiation.cc:407] Connection header received
0929 19:02:15.836004 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.836004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.836013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.836051 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:15.836212 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.836644 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.837415 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.837582 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.837631 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.838342 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:15.838343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:15.838344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:15.838348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:15.838365 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:15.838369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:15.838425 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.838476 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.838497 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:02:15.840065 20049 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:15.841549 20049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:15.847918 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.843948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.0.98.2:41023)
0929 19:02:15.844151 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.844156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.844263 (+ 107us) server_negotiation.cc:407] Connection header received
0929 19:02:15.844449 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.844452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.844470 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.844547 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.844732 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.845435 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.846741 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.847516 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.847675 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.847788 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.847827 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260929 19:02:15.847930 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.843823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41023 (local address 127.0.0.1:38278)
0929 19:02:15.844154 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.844192 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.844270 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.844566 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.844569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.844573 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.844642 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.844645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.845525 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.845526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.846658 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.846660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.847602 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.847687 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.847701 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.847770 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.847821 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":80,"threads_started":1}
I20260929 19:02:15.848042 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.844019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32865 (local address 127.0.0.1:34830)
0929 19:02:15.844398 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.844407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.844460 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.844644 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.844645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.844647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.844693 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.844696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.845972 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.845973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.846953 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.846955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.847805 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.847866 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.847870 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.847928 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.847965 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":9,"thread_start_us":59,"threads_started":1}
I20260929 19:02:15.848282 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.844212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.98.1:32865)
0929 19:02:15.844393 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.844396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.844452 (+ 56us) server_negotiation.cc:407] Connection header received
0929 19:02:15.844554 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.844557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.844570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.844635 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.844774 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.845877 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847052 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.847619 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847687 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.847773 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.848163 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.848205 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260929 19:02:15.848492 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.843791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36201 (local address 127.0.0.1:32986)
0929 19:02:15.844152 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.844191 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.844282 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.844945 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.844946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.844948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.844980 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.844982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.846158 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.846159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.847229 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.847232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.848254 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.848255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.848313 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.848317 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.848362 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.848423 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":87,"threads_started":1}
I20260929 19:02:15.848771 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.843976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.0.98.3:36201)
0929 19:02:15.844742 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.844747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.844756 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:15.844789 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.844791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.844805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.844873 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.845054 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.846074 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.847341 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.848049 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.848103 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.848186 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.848661 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.848699 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":86,"threads_started":1}
I20260929 19:02:15.848973 20302 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f (DEFAULT_TABLE table=test-table [id=adf55fb2cd6f4f289686fa7d7879c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.848978 20171 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f (DEFAULT_TABLE table=test-table [id=adf55fb2cd6f4f289686fa7d7879c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.849339 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.849349 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.849295 20433 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f (DEFAULT_TABLE table=test-table [id=adf55fb2cd6f4f289686fa7d7879c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:15.849588 20433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:15.852043 20513 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: Bootstrap starting.
I20260929 19:02:15.852743 20513 tablet_bootstrap.cc:654] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.852779 20515 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Bootstrap starting.
I20260929 19:02:15.853086 20513 log.cc:824] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.853402 20515 tablet_bootstrap.cc:654] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.853688 20515 log.cc:824] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.853781 20513 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: No bootstrap required, opened a new log
I20260929 19:02:15.853847 20513 ts_tablet_manager.cc:1397] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:15.854005 20514 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: Bootstrap starting.
I20260929 19:02:15.854672 20514 tablet_bootstrap.cc:654] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:15.854974 20514 log.cc:824] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:15.855515 20515 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: No bootstrap required, opened a new log
I20260929 19:02:15.855594 20515 ts_tablet_manager.cc:1397] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:02:15.855737 20514 tablet_bootstrap.cc:492] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: No bootstrap required, opened a new log
I20260929 19:02:15.855809 20514 ts_tablet_manager.cc:1397] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:15.856441 20513 raft_consensus.cc:348] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.856638 20513 raft_consensus.cc:374] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.856676 20513 raft_consensus.cc:729] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7564f84710d743f28f37aa9c38578e9d, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.856813 20513 consensus_queue.cc:260] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.857066 20513 ts_tablet_manager.cc:1428] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:15.857102 20368 heartbeater.cc:503] Master 127.0.98.62:39823 was elected leader, sending a full tablet report...
I20260929 19:02:15.857877 20515 raft_consensus.cc:348] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.858057 20515 raft_consensus.cc:374] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.858101 20515 raft_consensus.cc:729] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1580f5150b88441f8d2b835c79a7d9f5, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.858229 20515 consensus_queue.cc:260] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.858332 20514 raft_consensus.cc:348] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.858489 20515 ts_tablet_manager.cc:1428] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:15.858510 20514 raft_consensus.cc:374] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:15.858503 20499 heartbeater.cc:503] Master 127.0.98.62:39823 was elected leader, sending a full tablet report...
I20260929 19:02:15.858552 20514 raft_consensus.cc:729] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59503aafcd4142fd9d945c1102c2a0b5, State: Initialized, Role: FOLLOWER
I20260929 19:02:15.858664 20514 consensus_queue.cc:260] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.858932 20514 ts_tablet_manager.cc:1428] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:15.858979 20237 heartbeater.cc:503] Master 127.0.98.62:39823 was elected leader, sending a full tablet report...
I20260929 19:02:15.879153 20519 raft_consensus.cc:482] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:15.879258 20519 raft_consensus.cc:504] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.879717 20519 leader_election.cc:288] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1580f5150b88441f8d2b835c79a7d9f5 (127.0.98.3:36201), 59503aafcd4142fd9d945c1102c2a0b5 (127.0.98.1:32865)
I20260929 19:02:15.883627 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.879978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57081 (local address 127.0.98.3:36201)
0929 19:02:15.880040 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.880042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.880052 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:15.880139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.880140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.880148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.880187 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.880401 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.881168 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.882660 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.883269 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.883356 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.883366 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.883519 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.883559 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:02:15.883627 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.880048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37075 (local address 127.0.98.1:32865)
0929 19:02:15.880111 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.880113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.880246 (+ 133us) server_negotiation.cc:407] Connection header received
0929 19:02:15.880305 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.880305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.880315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.880391 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:15.880461 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.881173 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.882660 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.883217 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.883338 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.883351 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.883519 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.883560 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:02:15.883635 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.879863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36201 (local address 127.0.98.2:57081)
0929 19:02:15.879965 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.879993 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.880050 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.880219 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.880220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.880222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.880297 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.880324 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.881260 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.881261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.882560 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.882563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.883367 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.883367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.883434 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.883439 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.883501 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.883568 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":66}
I20260929 19:02:15.883641 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.879975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32865 (local address 127.0.98.2:37075)
0929 19:02:15.880140 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.880149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:15.880236 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:15.880360 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:15.880361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:15.880363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:15.880419 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.880420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.881260 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.881261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.882560 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:15.882563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.883353 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.883355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:15.883434 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.883439 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:02:15.883501 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:15.883550 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":58,"thread_start_us":88,"threads_started":1}
I20260929 19:02:15.884047 20453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcee8b7612e4e0a9d65eb0321e7ea6f" candidate_uuid: "7564f84710d743f28f37aa9c38578e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1580f5150b88441f8d2b835c79a7d9f5" is_pre_election: true
I20260929 19:02:15.884187 20453 raft_consensus.cc:2454] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7564f84710d743f28f37aa9c38578e9d in term 0.
I20260929 19:02:15.884254 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcee8b7612e4e0a9d65eb0321e7ea6f" candidate_uuid: "7564f84710d743f28f37aa9c38578e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59503aafcd4142fd9d945c1102c2a0b5" is_pre_election: true
I20260929 19:02:15.884428 20255 leader_election.cc:302] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1580f5150b88441f8d2b835c79a7d9f5, 7564f84710d743f28f37aa9c38578e9d; no voters:
I20260929 19:02:15.884439 20191 raft_consensus.cc:2454] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7564f84710d743f28f37aa9c38578e9d in term 0.
I20260929 19:02:15.884554 20519 raft_consensus.cc:2781] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:15.884632 20519 raft_consensus.cc:482] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:15.884665 20519 raft_consensus.cc:3037] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.885403 20519 raft_consensus.cc:504] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.885540 20519 leader_election.cc:288] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [CANDIDATE]: Term 1 election: Requested vote from peers 1580f5150b88441f8d2b835c79a7d9f5 (127.0.98.3:36201), 59503aafcd4142fd9d945c1102c2a0b5 (127.0.98.1:32865)
I20260929 19:02:15.885727 20453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcee8b7612e4e0a9d65eb0321e7ea6f" candidate_uuid: "7564f84710d743f28f37aa9c38578e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1580f5150b88441f8d2b835c79a7d9f5"
I20260929 19:02:15.885775 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcee8b7612e4e0a9d65eb0321e7ea6f" candidate_uuid: "7564f84710d743f28f37aa9c38578e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59503aafcd4142fd9d945c1102c2a0b5"
I20260929 19:02:15.885829 20453 raft_consensus.cc:3037] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.885861 20191 raft_consensus.cc:3037] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:15.886523 20453 raft_consensus.cc:2454] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7564f84710d743f28f37aa9c38578e9d in term 1.
I20260929 19:02:15.886689 20191 raft_consensus.cc:2454] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7564f84710d743f28f37aa9c38578e9d in term 1.
I20260929 19:02:15.886790 20255 leader_election.cc:302] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1580f5150b88441f8d2b835c79a7d9f5, 7564f84710d743f28f37aa9c38578e9d; no voters:
I20260929 19:02:15.886885 20519 raft_consensus.cc:2781] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:15.887079 20519 raft_consensus.cc:686] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [term 1 LEADER]: Becoming Leader. State: Replica: 7564f84710d743f28f37aa9c38578e9d, State: Running, Role: LEADER
I20260929 19:02:15.887183 20519 consensus_queue.cc:237] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } }
I20260929 19:02:15.888064 20049 catalog_manager.cc:5495] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7564f84710d743f28f37aa9c38578e9d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7564f84710d743f28f37aa9c38578e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7564f84710d743f28f37aa9c38578e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41023 } health_report { overall_health: HEALTHY } } }
I20260929 19:02:15.896215 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.893725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.0.98.2:41023)
0929 19:02:15.893789 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:15.893791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:15.893799 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:15.893920 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:15.893921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:15.893935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:15.893994 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:15.894171 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.894636 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.895443 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:15.895625 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:15.895682 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:15.895938 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:15.896029 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:15.896092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:15.896132 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 19:02:15.896147 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:02:15.898159 20453 raft_consensus.cc:1261] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 1 FOLLOWER]: Refusing update from remote peer 7564f84710d743f28f37aa9c38578e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:15.898159 20191 raft_consensus.cc:1261] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 59503aafcd4142fd9d945c1102c2a0b5 [term 1 FOLLOWER]: Refusing update from remote peer 7564f84710d743f28f37aa9c38578e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:15.898507 20523 consensus_queue.cc:1035] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "59503aafcd4142fd9d945c1102c2a0b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:15.898648 20519 consensus_queue.cc:1035] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 7564f84710d743f28f37aa9c38578e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1580f5150b88441f8d2b835c79a7d9f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:15.900960 20526 mvcc.cc:205] Tried to move back new op lower bound from 7334742163035795456 to 7334742162994872320. Current Snapshot: MvccSnapshot[applied={T|T < 7334742163035795456}]
I20260929 19:02:15.901551 20528 mvcc.cc:205] Tried to move back new op lower bound from 7334742163035795456 to 7334742162994872320. Current Snapshot: MvccSnapshot[applied={T|T < 7334742163035795456 or (T in {7334742163035795456})}]
I20260929 19:02:15.902118 20527 mvcc.cc:205] Tried to move back new op lower bound from 7334742163035795456 to 7334742162994872320. Current Snapshot: MvccSnapshot[applied={T|T < 7334742163035795456 or (T in {7334742163035795456})}]
I20260929 19:02:15.904080 20048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:15.904192 20048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:15.905016 20048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ceb89194bbf4960b83ab85286e71ccd: Sending DeleteTablet for 3 replicas of tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f
I20260929 19:02:15.905380 20302 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:15 UTC) from {username='slave'} at 127.0.0.1:38278
I20260929 19:02:15.905467 20433 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:15 UTC) from {username='slave'} at 127.0.0.1:32986
I20260929 19:02:15.905647 20536 tablet_replica.cc:330] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: stopping tablet replica
I20260929 19:02:15.905750 20536 raft_consensus.cc:2229] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:15.905742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20109
I20260929 19:02:15.906332 20536 raft_consensus.cc:2258] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:15.908155 20536 ts_tablet_manager.cc:1905] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:15.909999 20536 ts_tablet_manager.cc:1918] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:15.910070 20536 log.cc:1196] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Deleting WAL directory at /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708293967505-392-0/minicluster-data/ts-2/wal/wals/5fcee8b7612e4e0a9d65eb0321e7ea6f
I20260929 19:02:15.910331 20536 ts_tablet_manager.cc:1939] T 5fcee8b7612e4e0a9d65eb0321e7ea6f P 1580f5150b88441f8d2b835c79a7d9f5: Deleting consensus metadata
I20260929 19:02:15.910763 20033 catalog_manager.cc:4841] TS 1580f5150b88441f8d2b835c79a7d9f5 (127.0.98.3:36201): tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f (table test-table [id=adf55fb2cd6f4f289686fa7d7879c09d]) successfully deleted
W20260929 19:02:15.911674 20033 connection.cc:535] client connection to 127.0.98.1:32865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 19:02:15.911772 20033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:02:15.911841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20240
I20260929 19:02:15.913334 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.912466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32865 (local address 127.0.0.1:34832)
0929 19:02:15.913237 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.913298 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745}
W20260929 19:02:15.913535 20033 catalog_manager.cc:4584] TS 59503aafcd4142fd9d945c1102c2a0b5 (127.0.98.1:32865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fcee8b7612e4e0a9d65eb0321e7ea6f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32865: connect: Connection refused (error 111)
I20260929 19:02:15.917033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20371
I20260929 19:02:15.917078 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:15.916957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41023 (local address 127.0.0.1:38290)
0929 19:02:15.917014 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:15.917041 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:02:15.922057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20018
2026-09-29T19:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (668 ms)
[----------] 6 tests from EncryptionPolicyTest (8345 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:15 dist-test-slave-gfsd krb5kdc[20540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:15 dist-test-slave-gfsd krb5kdc[20540](info): set up 2 sockets
Sep 29 19:02:15 dist-test-slave-gfsd krb5kdc[20540](info): commencing operation
krb5kdc: starting...
W20260929 19:02:16.971663 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:17.094252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:17.175122 20553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:17.175381 20553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:17.175410 20553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:17.176692 20553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:17.176807 20553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:17.176854 20553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:17.176882 20553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:17.176906 20553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:17.178839 20553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:17.179054 20553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:17.179234 20553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:17.184227 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:17.184237 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:17.184285 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:17.184473 20553 server_base.cc:1034] running on GCE node
I20260929 19:02:17.184697 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:17.185009 20553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:17.186154 20553 hybrid_clock.cc:648] HybridClock initialized: now 1790708537186136 us; error 62 us; skew 500 ppm
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:17.188822 20553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:17.189282 20553 webserver.cc:460] Webserver started at http://127.0.0.1:42459/ using document root <none> and password file <none>
I20260929 19:02:17.189538 20553 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:17.189610 20553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:17.189759 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:17.190855 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "ae954ab2712d453db9130c52b286caf4"
format_stamp: "Formatted at 2026-09-29 19:02:17 on dist-test-slave-gfsd"
server_key: "092b430f22045bf85011a087d70e1685"
server_key_iv: "50bf4edc73a84a12bbef79e1a22bf98e"
server_key_version: "encryptionkey@0"
I20260929 19:02:17.191208 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae954ab2712d453db9130c52b286caf4"
format_stamp: "Formatted at 2026-09-29 19:02:17 on dist-test-slave-gfsd"
server_key: "092b430f22045bf85011a087d70e1685"
server_key_iv: "50bf4edc73a84a12bbef79e1a22bf98e"
server_key_version: "encryptionkey@0"
I20260929 19:02:17.192572 20553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:17.193341 20568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:17.193535 20553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:02:17.193640 20553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "ae954ab2712d453db9130c52b286caf4"
format_stamp: "Formatted at 2026-09-29 19:02:17 on dist-test-slave-gfsd"
server_key: "092b430f22045bf85011a087d70e1685"
server_key_iv: "50bf4edc73a84a12bbef79e1a22bf98e"
server_key_version: "encryptionkey@0"
I20260929 19:02:17.193712 20553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:17.207625 20553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:17.209606 20553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:17.209769 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:17.213562 20553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45913
I20260929 19:02:17.213629 20620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45913 every 8 connection(s)
I20260929 19:02:17.213964 20553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:17.214938 20621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:17.217459 20621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4: Bootstrap starting.
I20260929 19:02:17.218042 20621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:17.218302 20621 log.cc:824] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:17.219067 20621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4: No bootstrap required, opened a new log
I20260929 19:02:17.219172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20553
I20260929 19:02:17.219302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:17.219518 392 external_mini_cluster.cc:1428] Setting key 23016925082e71d27a3b8aadfd243caf
I20260929 19:02:17.221766 20621 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } }
I20260929 19:02:17.221969 20621 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:17.222014 20621 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae954ab2712d453db9130c52b286caf4, State: Initialized, Role: FOLLOWER
I20260929 19:02:17.222211 20621 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } }
I20260929 19:02:17.222304 20621 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:17.222349 20621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:17.222400 20621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:17.223179 20621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } }
I20260929 19:02:17.223356 20621 leader_election.cc:302] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae954ab2712d453db9130c52b286caf4; no voters:
I20260929 19:02:17.223537 20621 leader_election.cc:288] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:17.223676 20626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:17.223817 20621 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:17.223834 20626 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [term 1 LEADER]: Becoming Leader. State: Replica: ae954ab2712d453db9130c52b286caf4, State: Running, Role: LEADER
I20260929 19:02:17.224022 20626 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } }
I20260929 19:02:17.224429 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae954ab2712d453db9130c52b286caf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } } }
I20260929 19:02:17.224534 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:17.224671 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae954ab2712d453db9130c52b286caf4. Latest consensus state: current_term: 1 leader_uuid: "ae954ab2712d453db9130c52b286caf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae954ab2712d453db9130c52b286caf4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45913 } } }
I20260929 19:02:17.224728 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:17.224803 20635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:17.225361 20635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:17.227430 20635 catalog_manager.cc:1347] Generated new cluster ID: 0c7888b066b94023a27d9e5205903787
I20260929 19:02:17.227483 20635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:02:17.232852 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:17.220136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42585 (local address 127.0.0.1:45913)
0929 19:02:17.220325 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:17.220336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:17.220374 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:02:17.220868 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:17.220877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:17.220943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:17.221111 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:17.221755 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:17.223728 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:17.225151 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:17.225904 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:17.226025 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:17.229018 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:17.229022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:17.229022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:17.229033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:17.231054 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:17.231591 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:17.231593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:17.231594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:17.231696 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:17.231946 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:17.231946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:17.231947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:17.232232 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:17.232400 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:17.232554 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 19:02:17.232646 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260929 19:02:17.233897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:17.242333 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:17.235302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36995 (local address 127.0.0.1:45913)
0929 19:02:17.235339 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:17.235341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:17.235490 (+ 149us) server_negotiation.cc:407] Connection header received
0929 19:02:17.235569 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:17.235570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:17.235578 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:17.235604 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:17.236249 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:17.236986 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:17.237936 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:17.238261 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:17.238295 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:17.239692 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:17.239694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:17.239694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:17.239700 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:17.241165 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:17.241555 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:17.241556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:17.241556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:17.241602 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:17.241787 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:17.241787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:17.241788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:17.241906 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:17.241973 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:17.242169 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 19:02:17.242225 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:17.279549 20635 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:17.280150 20635 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:17.286434 20635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae954ab2712d453db9130c52b286caf4: Generated new TSK 0
I20260929 19:02:17.286623 20635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:17.301185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20553
W20260929 19:02:17.305898 20538 connection.cc:535] client connection to 127.0.0.1:45913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45913): Connection reset by peer (error 104)
2026-09-29T19:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20652](info): set up 2 sockets
Sep 29 19:02:17 dist-test-slave-gfsd krb5kdc[20652](info): commencing operation
krb5kdc: starting...
W20260929 19:02:18.339741 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:18 dist-test-slave-gfsd krb5kdc[20652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:18.456171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:18.537604 20665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:18.537801 20665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:18.537823 20665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:18.538975 20665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:18.539016 20665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:18.539031 20665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:18.539044 20665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:18.539096 20665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:18.540669 20665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:18.540920 20665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:18.541150 20665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:18.546154 20673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:18.546159 20671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:18.546175 20665 server_base.cc:1034] running on GCE node
W20260929 19:02:18.546161 20670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:18.546746 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:18.546985 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:18.548127 20665 hybrid_clock.cc:648] HybridClock initialized: now 1790708538548119 us; error 30 us; skew 500 ppm
Sep 29 19:02:18 dist-test-slave-gfsd krb5kdc[20652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:18.554683 20665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:18.555164 20665 webserver.cc:460] Webserver started at http://127.0.0.1:42585/ using document root <none> and password file <none>
I20260929 19:02:18.555467 20665 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:18.555540 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:18.555694 20665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:18.556739 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "1520d5b1b5704937a80588db5ccbdc7a"
format_stamp: "Formatted at 2026-09-29 19:02:18 on dist-test-slave-gfsd"
server_key: "f0491bf03bf1c60e9074dc0505e0e444"
server_key_iv: "fb5f913283f77bf8bb78f03ec57399a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:18.557094 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "1520d5b1b5704937a80588db5ccbdc7a"
format_stamp: "Formatted at 2026-09-29 19:02:18 on dist-test-slave-gfsd"
server_key: "f0491bf03bf1c60e9074dc0505e0e444"
server_key_iv: "fb5f913283f77bf8bb78f03ec57399a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:18.558457 20665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:18.559209 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:18.559384 20665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:18.559500 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "1520d5b1b5704937a80588db5ccbdc7a"
format_stamp: "Formatted at 2026-09-29 19:02:18 on dist-test-slave-gfsd"
server_key: "f0491bf03bf1c60e9074dc0505e0e444"
server_key_iv: "fb5f913283f77bf8bb78f03ec57399a3"
server_key_version: "encryptionkey@0"
I20260929 19:02:18.559585 20665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:18.584774 20665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:18.586937 20665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:18.587109 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:18.591182 20665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38623
I20260929 19:02:18.591221 20732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38623 every 8 connection(s)
I20260929 19:02:18.591612 20665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:18.592456 20733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:18.595196 20733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a: Bootstrap starting.
I20260929 19:02:18.595947 20733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:18.596262 20733 log.cc:824] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:18.597010 20733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a: No bootstrap required, opened a new log
I20260929 19:02:18.598543 20733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260929 19:02:18.598676 20733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:18.598698 20733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1520d5b1b5704937a80588db5ccbdc7a, State: Initialized, Role: FOLLOWER
I20260929 19:02:18.598840 20733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260929 19:02:18.598923 20733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:18.598946 20733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:18.598973 20733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:18.599467 20733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260929 19:02:18.599560 20733 leader_election.cc:302] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1520d5b1b5704937a80588db5ccbdc7a; no voters:
I20260929 19:02:18.599677 20733 leader_election.cc:288] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:18.599757 20736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:18.599908 20736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [term 1 LEADER]: Becoming Leader. State: Replica: 1520d5b1b5704937a80588db5ccbdc7a, State: Running, Role: LEADER
I20260929 19:02:18.599977 20733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:18.600039 20736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260929 19:02:18.600440 20738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1520d5b1b5704937a80588db5ccbdc7a. Latest consensus state: current_term: 1 leader_uuid: "1520d5b1b5704937a80588db5ccbdc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } } }
I20260929 19:02:18.600524 20738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:18.600654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20665
I20260929 19:02:18.600670 20737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1520d5b1b5704937a80588db5ccbdc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1520d5b1b5704937a80588db5ccbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } } }
I20260929 19:02:18.600741 20737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:18.600786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:18.601027 392 external_mini_cluster.cc:1428] Setting key da6331da11dbec24ba5ef62f2fcace6e
I20260929 19:02:18.601339 20741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:18.601894 20741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:18.603686 20741 catalog_manager.cc:1347] Generated new cluster ID: 6e0631d871a64cbd92a9f13dd65609dd
I20260929 19:02:18.603739 20741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:18.607465 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:18.601571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39915 (local address 127.0.0.1:38623)
0929 19:02:18.601803 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:18.601814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:18.602073 (+ 259us) server_negotiation.cc:407] Connection header received
0929 19:02:18.602280 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:18.602287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:18.602341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:18.602456 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:18.602790 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:18.604149 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:18.605250 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:18.605874 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:18.605990 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:18.606732 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:18.606736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:18.606736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:18.606944 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:18.606983 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:18.606998 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:18.607087 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:18.607170 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 19:02:18.607199 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"spinlock_wait_cycles":2816,"thread_start_us":107,"threads_started":1}
I20260929 19:02:18.608311 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:18.613286 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:18.609669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52101 (local address 127.0.0.1:38623)
0929 19:02:18.609735 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:18.609738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:18.609852 (+ 114us) server_negotiation.cc:407] Connection header received
0929 19:02:18.609906 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:18.609907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:18.609917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:18.609973 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:18.610117 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:18.610864 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:18.611924 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:18.612236 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:18.612303 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:18.613034 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:18.613035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:18.613035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:18.613039 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:18.613053 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:18.613057 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:18.613112 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:18.613206 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 19:02:18.613218 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:02:18.633992 20741 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:18.634533 20741 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:18.639758 20741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1520d5b1b5704937a80588db5ccbdc7a: Generated new TSK 0
I20260929 19:02:18.639962 20741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:18.647737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20665
W20260929 19:02:18.651882 20650 connection.cc:535] client connection to 127.0.0.1:38623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38623): Connection reset by peer (error 104)
2026-09-29T19:02:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:18 dist-test-slave-gfsd krb5kdc[20764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:18 dist-test-slave-gfsd krb5kdc[20764](info): set up 2 sockets
Sep 29 19:02:18 dist-test-slave-gfsd krb5kdc[20764](info): commencing operation
krb5kdc: starting...
W20260929 19:02:19.687179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:19 dist-test-slave-gfsd krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:19.806260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:19.887557 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:19.887786 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:19.887832 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:19.888979 20777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:19.889045 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:19.889070 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:19.889101 20777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:19.889127 20777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:19.890767 20777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:19.891045 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:19.891268 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:19.896095 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:19.896112 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:19.896377 20777 server_base.cc:1034] running on GCE node
W20260929 19:02:19.896095 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:19.896708 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:19.897009 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:19.898170 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790708539898157 us; error 46 us; skew 500 ppm
Sep 29 19:02:19 dist-test-slave-gfsd krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:19.900648 20777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:19.901065 20777 webserver.cc:460] Webserver started at http://127.0.0.1:44987/ using document root <none> and password file <none>
I20260929 19:02:19.901291 20777 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:19.901356 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:19.901511 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:19.902560 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "e23b890450c741e291ecaceecfe29ddb"
format_stamp: "Formatted at 2026-09-29 19:02:19 on dist-test-slave-gfsd"
server_key: "f4c117ef7748ce9b3992e20c7570022c"
server_key_iv: "0c8a38807ecd6d3561759349a9351cbb"
server_key_version: "encryptionkey@0"
I20260929 19:02:19.902956 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "e23b890450c741e291ecaceecfe29ddb"
format_stamp: "Formatted at 2026-09-29 19:02:19 on dist-test-slave-gfsd"
server_key: "f4c117ef7748ce9b3992e20c7570022c"
server_key_iv: "0c8a38807ecd6d3561759349a9351cbb"
server_key_version: "encryptionkey@0"
I20260929 19:02:19.904480 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:19.905303 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:19.905596 20777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:19.905705 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "e23b890450c741e291ecaceecfe29ddb"
format_stamp: "Formatted at 2026-09-29 19:02:19 on dist-test-slave-gfsd"
server_key: "f4c117ef7748ce9b3992e20c7570022c"
server_key_iv: "0c8a38807ecd6d3561759349a9351cbb"
server_key_version: "encryptionkey@0"
I20260929 19:02:19.905851 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:19.909262 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:19.911525 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:19.911671 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:19.915607 20777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36275
I20260929 19:02:19.915637 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36275 every 8 connection(s)
I20260929 19:02:19.915969 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:19.916850 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:19.919034 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: Bootstrap starting.
I20260929 19:02:19.919663 20845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:19.919945 20845 log.cc:824] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:19.920626 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: No bootstrap required, opened a new log
I20260929 19:02:19.920933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20777
I20260929 19:02:19.921056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:19.921247 392 external_mini_cluster.cc:1428] Setting key deeb3dc55d62e4b113b8c8265f5a2806
I20260929 19:02:19.922727 20845 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260929 19:02:19.922897 20845 raft_consensus.cc:374] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:19.922942 20845 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23b890450c741e291ecaceecfe29ddb, State: Initialized, Role: FOLLOWER
I20260929 19:02:19.923130 20845 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260929 19:02:19.923249 20845 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:19.923363 20845 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:19.923449 20845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:19.924046 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:19.921746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36977 (local address 127.0.0.1:36275)
0929 19:02:19.921979 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:19.921986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:19.922010 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:02:19.922710 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:19.922715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:19.922740 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:19.922850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:19.923612 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:19.923615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:19.923616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:19.923620 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:19.923669 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:19.923680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:19.923757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:19.923829 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:02:19.923877 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":152,"threads_started":1}
I20260929 19:02:19.924477 20845 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260929 19:02:19.924620 20845 leader_election.cc:302] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23b890450c741e291ecaceecfe29ddb; no voters:
I20260929 19:02:19.924815 20845 leader_election.cc:288] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:19.924841 20850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:19.925034 20850 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [term 1 LEADER]: Becoming Leader. State: Replica: e23b890450c741e291ecaceecfe29ddb, State: Running, Role: LEADER
I20260929 19:02:19.925120 20845 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:19.925134 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:19.925156 20850 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260929 19:02:19.925472 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23b890450c741e291ecaceecfe29ddb. Latest consensus state: current_term: 1 leader_uuid: "e23b890450c741e291ecaceecfe29ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260929 19:02:19.925488 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23b890450c741e291ecaceecfe29ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23b890450c741e291ecaceecfe29ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260929 19:02:19.925681 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:19.925868 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:19.927088 20869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:19.927143 20869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:19.927196 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:19.927415 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:19.928359 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:19.927081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.0.0.1:36275)
0929 19:02:19.927113 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:19.927114 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:19.927515 (+ 401us) server_negotiation.cc:407] Connection header received
0929 19:02:19.927539 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:19.927540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:19.927546 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:19.927609 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:19.928132 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:19.928133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:19.928133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:19.928137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:19.928151 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:19.928156 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:19.928217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:19.928292 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:02:19.928305 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 19:02:19.929037 20856 catalog_manager.cc:1347] Generated new cluster ID: 36846b23b0404fe7ba1db3195018720e
I20260929 19:02:19.929100 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:19.962724 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:19.962932 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:19.971099 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23b890450c741e291ecaceecfe29ddb: Generated new TSK 0
I20260929 19:02:19.971372 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:19.976328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20777
2026-09-29T19:02:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:20 dist-test-slave-gfsd krb5kdc[20876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:20 dist-test-slave-gfsd krb5kdc[20876](info): set up 2 sockets
Sep 29 19:02:20 dist-test-slave-gfsd krb5kdc[20876](info): commencing operation
krb5kdc: starting...
W20260929 19:02:21.019362 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:21.137913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:21.217911 20889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:21.218107 20889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:21.218128 20889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:21.219269 20889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:21.219336 20889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:21.219352 20889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:21.219363 20889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:21.219375 20889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:21.220865 20889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:21.221110 20889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:21.221334 20889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:21.226554 20897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:21.226572 20894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:21.226567 20895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:21.226725 20889 server_base.cc:1034] running on GCE node
I20260929 19:02:21.227126 20889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:21.227460 20889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:21.228633 20889 hybrid_clock.cc:648] HybridClock initialized: now 1790708541228627 us; error 37 us; skew 500 ppm
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:21.231215 20889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:21.231642 20889 webserver.cc:460] Webserver started at http://127.0.0.1:39391/ using document root <none> and password file <none>
I20260929 19:02:21.231837 20889 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:21.231877 20889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:21.231964 20889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:21.233007 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13"
format_stamp: "Formatted at 2026-09-29 19:02:21 on dist-test-slave-gfsd"
server_key: "0bcb10810989fe5db3dc8b9d2c2b548b"
server_key_iv: "fd310cc17ba186f2654d854c6026aa24"
server_key_version: "encryptionkey@0"
I20260929 19:02:21.233361 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13"
format_stamp: "Formatted at 2026-09-29 19:02:21 on dist-test-slave-gfsd"
server_key: "0bcb10810989fe5db3dc8b9d2c2b548b"
server_key_iv: "fd310cc17ba186f2654d854c6026aa24"
server_key_version: "encryptionkey@0"
I20260929 19:02:21.234678 20889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:21.235885 20904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:21.236088 20889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:21.236181 20889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13"
format_stamp: "Formatted at 2026-09-29 19:02:21 on dist-test-slave-gfsd"
server_key: "0bcb10810989fe5db3dc8b9d2c2b548b"
server_key_iv: "fd310cc17ba186f2654d854c6026aa24"
server_key_version: "encryptionkey@0"
I20260929 19:02:21.236233 20889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:21.249588 20889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:21.251396 20889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:21.251511 20889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:21.255093 20889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37651
I20260929 19:02:21.255148 20956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37651 every 8 connection(s)
I20260929 19:02:21.255501 20889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:21.256340 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:21.258510 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13: Bootstrap starting.
I20260929 19:02:21.259075 20957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:21.259382 20957 log.cc:824] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:21.260100 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13: No bootstrap required, opened a new log
I20260929 19:02:21.262097 20957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } }
I20260929 19:02:21.262257 20957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:21.262302 20957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f00ccdc5b2b4175aa5a88bb32a65d13, State: Initialized, Role: FOLLOWER
I20260929 19:02:21.262472 20957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } }
I20260929 19:02:21.262557 20957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:21.262598 20957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:21.262648 20957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:21.262692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 20889
I20260929 19:02:21.262907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:21.263118 392 external_mini_cluster.cc:1428] Setting key 21e13aab23a3d47799f6a1b706017ea1
I20260929 19:02:21.263348 20957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } }
I20260929 19:02:21.263471 20957 leader_election.cc:302] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f00ccdc5b2b4175aa5a88bb32a65d13; no voters:
I20260929 19:02:21.263643 20957 leader_election.cc:288] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:21.263809 20962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:21.264031 20962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [term 1 LEADER]: Becoming Leader. State: Replica: 7f00ccdc5b2b4175aa5a88bb32a65d13, State: Running, Role: LEADER
I20260929 19:02:21.264036 20957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:21.264261 20962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } }
I20260929 19:02:21.264631 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } } }
I20260929 19:02:21.264623 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f00ccdc5b2b4175aa5a88bb32a65d13. Latest consensus state: current_term: 1 leader_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f00ccdc5b2b4175aa5a88bb32a65d13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37651 } } }
I20260929 19:02:21.264716 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:21.264724 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:21.264950 20966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:21.265508 20966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:21.267024 20966 catalog_manager.cc:1347] Generated new cluster ID: e431ff74d3a1415180978ad3fd7b280c
I20260929 19:02:21.267073 20966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:02:21.274534 20966 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:21.275243 20966 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:21.275856 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:21.263738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35577 (local address 127.0.0.1:37651)
0929 19:02:21.264011 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:21.264021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:21.264059 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:02:21.264192 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:21.264199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:21.264260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:21.264427 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:21.265502 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:21.267046 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:21.268186 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:21.268801 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:21.268932 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:21.271581 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:21.271585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:21.271586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:21.271600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:21.274267 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:21.274702 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:21.274705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:21.274706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:21.274798 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:21.275011 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:21.275012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:21.275012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:21.275230 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:21.275432 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:21.275502 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 19:02:21.275651 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260929 19:02:21.276721 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:21.281054 20966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f00ccdc5b2b4175aa5a88bb32a65d13: Generated new TSK 0
I20260929 19:02:21.281267 20966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:21.284322 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:21.278029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39897 (local address 127.0.0.1:37651)
0929 19:02:21.278065 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:21.278067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:21.278169 (+ 102us) server_negotiation.cc:407] Connection header received
0929 19:02:21.278225 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:21.278225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:21.278235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:21.278290 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:21.278901 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:21.279449 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:21.280181 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:21.280404 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:21.280487 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:21.281717 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:21.281719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:21.281719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:21.281726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:21.283123 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:21.283554 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:21.283555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:21.283555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:21.283615 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:21.283822 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:21.283823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:21.283823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:21.283933 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:21.284035 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:21.284142 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:02:21.284219 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:21.286088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 20889
2026-09-29T19:02:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20988](info): set up 2 sockets
Sep 29 19:02:21 dist-test-slave-gfsd krb5kdc[20988](info): commencing operation
krb5kdc: starting...
W20260929 19:02:22.323434 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:22 dist-test-slave-gfsd krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:22.442487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:22.522089 21001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:22.522317 21001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:22.522356 21001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:22.523530 21001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:22.523597 21001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:22.523623 21001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:22.523656 21001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:22.523679 21001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:22.525593 21001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:22.525876 21001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:22.526124 21001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:22.530918 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:22.531080 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:22.531190 21009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:22.531086 21001 server_base.cc:1034] running on GCE node
I20260929 19:02:22.531477 21001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:22.531719 21001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:22.532857 21001 hybrid_clock.cc:648] HybridClock initialized: now 1790708542532810 us; error 66 us; skew 500 ppm
Sep 29 19:02:22 dist-test-slave-gfsd krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:22.535403 21001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:22.535846 21001 webserver.cc:460] Webserver started at http://127.0.0.1:40091/ using document root <none> and password file <none>
I20260929 19:02:22.536032 21001 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:22.536073 21001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:22.536163 21001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:22.537147 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "feec3fa068944d54897e7e751105bd97"
format_stamp: "Formatted at 2026-09-29 19:02:22 on dist-test-slave-gfsd"
server_key: "5e5d6d3aa2ba5a1d6cc3c52f7cdd099f"
server_key_iv: "d5385948c8ccc77709d7334383aabec9"
server_key_version: "encryptionkey@0"
I20260929 19:02:22.537438 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "feec3fa068944d54897e7e751105bd97"
format_stamp: "Formatted at 2026-09-29 19:02:22 on dist-test-slave-gfsd"
server_key: "5e5d6d3aa2ba5a1d6cc3c52f7cdd099f"
server_key_iv: "d5385948c8ccc77709d7334383aabec9"
server_key_version: "encryptionkey@0"
I20260929 19:02:22.538637 21001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:22.539458 21016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:22.539655 21001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:22.539752 21001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "feec3fa068944d54897e7e751105bd97"
format_stamp: "Formatted at 2026-09-29 19:02:22 on dist-test-slave-gfsd"
server_key: "5e5d6d3aa2ba5a1d6cc3c52f7cdd099f"
server_key_iv: "d5385948c8ccc77709d7334383aabec9"
server_key_version: "encryptionkey@0"
I20260929 19:02:22.539888 21001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:22.568495 21001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:22.570623 21001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:22.570785 21001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:22.574627 21001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34039
I20260929 19:02:22.574699 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34039 every 8 connection(s)
I20260929 19:02:22.575043 21001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:22.575994 21069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:22.577039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21001
I20260929 19:02:22.577160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:22.577369 392 external_mini_cluster.cc:1428] Setting key 747747108890703746e9ef0556f723b5
I20260929 19:02:22.579101 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: Bootstrap starting.
I20260929 19:02:22.579886 21069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:22.580224 21069 log.cc:824] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:22.581023 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: No bootstrap required, opened a new log
I20260929 19:02:22.583663 21069 raft_consensus.cc:348] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } }
I20260929 19:02:22.583884 21069 raft_consensus.cc:374] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:22.583925 21069 raft_consensus.cc:729] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feec3fa068944d54897e7e751105bd97, State: Initialized, Role: FOLLOWER
I20260929 19:02:22.584107 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:22.577881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48105 (local address 127.0.0.1:34039)
0929 19:02:22.578091 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:22.578101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:22.578135 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:02:22.578751 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:22.578762 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:22.578842 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:22.579002 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:22.579183 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:22.580720 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:22.581633 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:22.582335 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:22.582460 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:22.583249 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:22.583256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:22.583256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:22.583507 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:22.583549 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:22.583565 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:22.583643 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:22.583769 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 19:02:22.583844 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260929 19:02:22.584072 21069 consensus_queue.cc:260] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } }
I20260929 19:02:22.584179 21069 raft_consensus.cc:388] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:22.584218 21069 raft_consensus.cc:482] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:22.584265 21069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:22.585076 21069 raft_consensus.cc:504] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } }
I20260929 19:02:22.585098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:22.585217 21069 leader_election.cc:302] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feec3fa068944d54897e7e751105bd97; no voters:
I20260929 19:02:22.585462 21069 leader_election.cc:288] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:22.585493 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:22.585614 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [term 1 LEADER]: Becoming Leader. State: Replica: feec3fa068944d54897e7e751105bd97, State: Running, Role: LEADER
I20260929 19:02:22.585704 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } }
I20260929 19:02:22.585737 21069 sys_catalog.cc:564] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:22.586071 21074 sys_catalog.cc:455] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feec3fa068944d54897e7e751105bd97. Latest consensus state: current_term: 1 leader_uuid: "feec3fa068944d54897e7e751105bd97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } } }
I20260929 19:02:22.586174 21074 sys_catalog.cc:458] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:22.586087 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feec3fa068944d54897e7e751105bd97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feec3fa068944d54897e7e751105bd97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34039 } } }
I20260929 19:02:22.586350 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97 [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:22.587641 21093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:22.587740 21093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:22.588057 21083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:22.588212 21083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:22.589731 21083 catalog_manager.cc:1347] Generated new cluster ID: 970fc4e1d4d14e478ceedc93766f412c
I20260929 19:02:22.589787 21083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:22.592557 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:22.587730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46783 (local address 127.0.0.1:34039)
0929 19:02:22.587764 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:22.587767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:22.588998 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 19:02:22.589029 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:22.589030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:22.589041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:22.589115 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:22.589383 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:22.590149 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:22.591175 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:22.591536 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:22.591601 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:22.592312 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:22.592313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:22.592313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:22.592318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:22.592332 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:22.592336 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:22.592392 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:22.592459 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:02:22.592483 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:22.598953 21083 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:22.599555 21083 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:22.604645 21083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feec3fa068944d54897e7e751105bd97: Generated new TSK 0
I20260929 19:02:22.604893 21083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:22.611286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21001
2026-09-29T19:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:22 dist-test-slave-gfsd krb5kdc[21100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:22 dist-test-slave-gfsd krb5kdc[21100](info): set up 2 sockets
Sep 29 19:02:22 dist-test-slave-gfsd krb5kdc[21100](info): commencing operation
krb5kdc: starting...
W20260929 19:02:23.643520 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:23 dist-test-slave-gfsd krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:23.763028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:23.844074 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:23.844280 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:23.844303 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:23.845408 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:23.845443 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:23.845458 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:23.845470 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:23.845482 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:23.846886 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:23.847141 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:23.847348 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:23.852155 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:23.852156 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:23.852144 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:23.852348 21113 server_base.cc:1034] running on GCE node
I20260929 19:02:23.852648 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:23.852948 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:23.854099 21113 hybrid_clock.cc:648] HybridClock initialized: now 1790708543854084 us; error 25 us; skew 500 ppm
Sep 29 19:02:23 dist-test-slave-gfsd krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:23.860146 21113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:23.860620 21113 webserver.cc:460] Webserver started at http://127.0.0.1:35117/ using document root <none> and password file <none>
I20260929 19:02:23.860872 21113 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:23.860944 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:23.861090 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:23.862172 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "e8e0492015b541959671ca5c7f9d1611"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-gfsd"
server_key: "ca11907fd06b7110bc4fda38770ce544"
server_key_iv: "10436e04cf6109a68ba92d9cd0b589ce"
server_key_version: "encryptionkey@0"
I20260929 19:02:23.862535 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8e0492015b541959671ca5c7f9d1611"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-gfsd"
server_key: "ca11907fd06b7110bc4fda38770ce544"
server_key_iv: "10436e04cf6109a68ba92d9cd0b589ce"
server_key_version: "encryptionkey@0"
I20260929 19:02:23.864017 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:02:23.865002 21128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:23.865298 21113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:23.865463 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "e8e0492015b541959671ca5c7f9d1611"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-gfsd"
server_key: "ca11907fd06b7110bc4fda38770ce544"
server_key_iv: "10436e04cf6109a68ba92d9cd0b589ce"
server_key_version: "encryptionkey@0"
I20260929 19:02:23.865547 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:23.868999 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:23.871125 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:23.871353 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:23.875502 21113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44585
I20260929 19:02:23.875548 21180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44585 every 8 connection(s)
I20260929 19:02:23.875874 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:23.876755 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:23.877617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21113
I20260929 19:02:23.877748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:23.877955 392 external_mini_cluster.cc:1428] Setting key e03bba55fa415b3a9665f0125d26cf6e
I20260929 19:02:23.879226 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: Bootstrap starting.
I20260929 19:02:23.879941 21181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:23.880266 21181 log.cc:824] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:23.880439 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:23.878477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44149 (local address 127.0.0.1:44585)
0929 19:02:23.878659 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:23.878670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:23.878707 (+ 37us) server_negotiation.cc:407] Connection header received
0929 19:02:23.879124 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:23.879130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:23.879153 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:23.879326 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:23.879985 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:23.879988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:23.879988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:23.879992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:23.880041 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:23.880055 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:23.880136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:23.880170 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 19:02:23.880208 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260929 19:02:23.881042 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: No bootstrap required, opened a new log
I20260929 19:02:23.881412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:23.882931 21181 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } }
I20260929 19:02:23.883118 21181 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:23.883172 21181 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e0492015b541959671ca5c7f9d1611, State: Initialized, Role: FOLLOWER
I20260929 19:02:23.883327 21181 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } }
I20260929 19:02:23.883416 21181 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:23.883452 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:23.883502 21181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:23.883961 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:23.882834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56095 (local address 127.0.0.1:44585)
0929 19:02:23.882897 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:23.882899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:23.882995 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:02:23.883047 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:23.883047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:23.883052 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:23.883112 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:23.883704 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:23.883705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:23.883705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:23.883709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:23.883721 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:23.883725 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:23.883793 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:23.883892 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:02:23.883905 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:02:23.884258 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } }
I20260929 19:02:23.884361 21181 leader_election.cc:302] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e0492015b541959671ca5c7f9d1611; no voters:
I20260929 19:02:23.884505 21181 leader_election.cc:288] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:23.884538 21191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:23.884681 21191 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [term 1 LEADER]: Becoming Leader. State: Replica: e8e0492015b541959671ca5c7f9d1611, State: Running, Role: LEADER
I20260929 19:02:23.884750 21181 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:23.884764 21191 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } }
I20260929 19:02:23.885156 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e0492015b541959671ca5c7f9d1611. Latest consensus state: current_term: 1 leader_uuid: "e8e0492015b541959671ca5c7f9d1611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } } }
I20260929 19:02:23.885164 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e0492015b541959671ca5c7f9d1611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e0492015b541959671ca5c7f9d1611" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44585 } } }
I20260929 19:02:23.885273 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:23.885294 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611 [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:23.886459 21207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:23.886523 21207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:23.886586 21199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:23.886759 21199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:23.888092 21199 catalog_manager.cc:1347] Generated new cluster ID: 3f9071095d1d4c91a54c592b5067b844
I20260929 19:02:23.888150 21199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:23.900913 21199 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:23.901081 21199 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:23.909226 21199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e0492015b541959671ca5c7f9d1611: Generated new TSK 0
I20260929 19:02:23.909406 21199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:23.913105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21113
2026-09-29T19:02:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:23 dist-test-slave-gfsd krb5kdc[21212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:23 dist-test-slave-gfsd krb5kdc[21212](info): set up 2 sockets
Sep 29 19:02:23 dist-test-slave-gfsd krb5kdc[21212](info): commencing operation
krb5kdc: starting...
W20260929 19:02:24.955487 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:25.082477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:25.162593 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:25.162799 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:25.162822 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:25.163918 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:25.163951 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:25.163965 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:25.163978 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:25.163990 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:25.165341 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:25.165521 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:25.165680 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:25.170758 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:25.170733 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:25.170894 21225 server_base.cc:1034] running on GCE node
W20260929 19:02:25.170733 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:25.171358 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:25.171667 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:25.172869 21225 hybrid_clock.cc:648] HybridClock initialized: now 1790708545172860 us; error 33 us; skew 500 ppm
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:25.175448 21225 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:25.175897 21225 webserver.cc:460] Webserver started at http://127.0.0.1:42505/ using document root <none> and password file <none>
I20260929 19:02:25.176122 21225 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:25.176165 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:25.176309 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:25.177429 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "d805599b727940279ce6be6cabdf31e3"
format_stamp: "Formatted at 2026-09-29 19:02:25 on dist-test-slave-gfsd"
server_key: "20a9bc74a98015f0c9db29f97960381d"
server_key_iv: "267a3cad29fe57db6f9efc4b11d3a059"
server_key_version: "encryptionkey@0"
I20260929 19:02:25.177780 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "d805599b727940279ce6be6cabdf31e3"
format_stamp: "Formatted at 2026-09-29 19:02:25 on dist-test-slave-gfsd"
server_key: "20a9bc74a98015f0c9db29f97960381d"
server_key_iv: "267a3cad29fe57db6f9efc4b11d3a059"
server_key_version: "encryptionkey@0"
I20260929 19:02:25.179091 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:25.179962 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:25.180130 21225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:25.180244 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "d805599b727940279ce6be6cabdf31e3"
format_stamp: "Formatted at 2026-09-29 19:02:25 on dist-test-slave-gfsd"
server_key: "20a9bc74a98015f0c9db29f97960381d"
server_key_iv: "267a3cad29fe57db6f9efc4b11d3a059"
server_key_version: "encryptionkey@0"
I20260929 19:02:25.180325 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:25.190726 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:25.192588 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:25.192752 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:25.196738 21225 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41171
I20260929 19:02:25.196771 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41171 every 8 connection(s)
I20260929 19:02:25.197180 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:25.198014 21293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:25.200378 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3: Bootstrap starting.
I20260929 19:02:25.200920 21293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:25.201150 21293 log.cc:824] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:25.201831 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3: No bootstrap required, opened a new log
I20260929 19:02:25.203424 21293 raft_consensus.cc:348] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:02:25.203563 21293 raft_consensus.cc:374] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:25.203595 21293 raft_consensus.cc:729] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d805599b727940279ce6be6cabdf31e3, State: Initialized, Role: FOLLOWER
I20260929 19:02:25.203743 21293 consensus_queue.cc:260] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:02:25.203835 21293 raft_consensus.cc:388] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:25.203864 21293 raft_consensus.cc:482] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:25.203894 21293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:25.204391 21293 raft_consensus.cc:504] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:02:25.204474 21293 leader_election.cc:302] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d805599b727940279ce6be6cabdf31e3; no voters:
I20260929 19:02:25.204598 21293 leader_election.cc:288] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:25.204677 21296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:25.204867 21293 sys_catalog.cc:564] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:25.204943 21296 raft_consensus.cc:686] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [term 1 LEADER]: Becoming Leader. State: Replica: d805599b727940279ce6be6cabdf31e3, State: Running, Role: LEADER
I20260929 19:02:25.205070 21296 consensus_queue.cc:237] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:02:25.205695 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d805599b727940279ce6be6cabdf31e3. Latest consensus state: current_term: 1 leader_uuid: "d805599b727940279ce6be6cabdf31e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 19:02:25.205778 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:25.205884 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d805599b727940279ce6be6cabdf31e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d805599b727940279ce6be6cabdf31e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 19:02:25.205950 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:25.206125 21305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:25.206727 21305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:25.207322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21225
I20260929 19:02:25.207453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:25.207648 392 external_mini_cluster.cc:1428] Setting key 0a83965e83aa3fdae3f103d3534a1237
I20260929 19:02:25.208518 21305 catalog_manager.cc:1347] Generated new cluster ID: 6c43d71da8d44b50a813c4267f604eb2
I20260929 19:02:25.208575 21305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:02:25.219879 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.208163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:43455 (local address 127.0.0.1:41171)
0929 19:02:25.208587 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.208598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.208714 (+ 116us) server_negotiation.cc:407] Connection header received
0929 19:02:25.208839 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.208846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.208903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.209039 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:25.209879 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.211274 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.212376 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.212983 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.213081 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.216068 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:25.216072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:25.216073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:25.216084 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:25.218124 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.218561 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.218565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.218565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.218677 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.218909 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.218910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.218910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.219226 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:25.219426 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.219533 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.219635 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":75,"threads_started":1}
I20260929 19:02:25.220882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:25.223254 21305 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:25.223948 21305 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:25.226992 21305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d805599b727940279ce6be6cabdf31e3: Generated new TSK 0
I20260929 19:02:25.227170 21305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:25.229974 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.222301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:46613 (local address 127.0.0.1:41171)
0929 19:02:25.222347 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.222349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.222463 (+ 114us) server_negotiation.cc:407] Connection header received
0929 19:02:25.222507 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.222508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.222517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.222569 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:25.223208 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.224016 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.224997 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.225312 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.225348 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.227322 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:25.227323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:25.227324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:25.227332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:25.228684 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.229094 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.229095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.229096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.229153 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.229421 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.229422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.229422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.229540 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:25.229658 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.229800 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.229867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:25.232090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21225
W20260929 19:02:25.236652 21210 connection.cc:535] client connection to 127.0.0.1:41171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41171): Connection reset by peer (error 104)
2026-09-29T19:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21324](info): set up 2 sockets
Sep 29 19:02:25 dist-test-slave-gfsd krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260929 19:02:26.263545 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:26 dist-test-slave-gfsd krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:26.382387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:26.462772 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:26.462999 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:26.463037 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:26.464208 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:26.464282 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:26.464310 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:26.464339 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:26.464362 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:26.465803 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:26.466068 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:26.466291 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:26.471136 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:26.471159 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:26.471179 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:26.471329 21337 server_base.cc:1034] running on GCE node
I20260929 19:02:26.471659 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:26.471949 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:26.473119 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790708546473102 us; error 30 us; skew 500 ppm
Sep 29 19:02:26 dist-test-slave-gfsd krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:26.475963 21337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:26.476459 21337 webserver.cc:460] Webserver started at http://127.0.0.1:37561/ using document root <none> and password file <none>
I20260929 19:02:26.476687 21337 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:26.476763 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:26.476912 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:26.477954 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "14b9cadaa3914aef9d41bcd158b7fc2e"
format_stamp: "Formatted at 2026-09-29 19:02:26 on dist-test-slave-gfsd"
server_key: "72c2294231de3ad364f33a9feb947349"
server_key_iv: "2dd39133d0037c7f1749b20f4335e337"
server_key_version: "encryptionkey@0"
I20260929 19:02:26.478314 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "14b9cadaa3914aef9d41bcd158b7fc2e"
format_stamp: "Formatted at 2026-09-29 19:02:26 on dist-test-slave-gfsd"
server_key: "72c2294231de3ad364f33a9feb947349"
server_key_iv: "2dd39133d0037c7f1749b20f4335e337"
server_key_version: "encryptionkey@0"
I20260929 19:02:26.479805 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:26.480562 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:26.480782 21337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:26.480893 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "14b9cadaa3914aef9d41bcd158b7fc2e"
format_stamp: "Formatted at 2026-09-29 19:02:26 on dist-test-slave-gfsd"
server_key: "72c2294231de3ad364f33a9feb947349"
server_key_iv: "2dd39133d0037c7f1749b20f4335e337"
server_key_version: "encryptionkey@0"
I20260929 19:02:26.480967 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:26.491411 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:26.493777 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:26.494014 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:26.498469 21337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38629
I20260929 19:02:26.498543 21404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38629 every 8 connection(s)
I20260929 19:02:26.498909 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:26.499946 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:26.502468 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: Bootstrap starting.
I20260929 19:02:26.503048 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:26.503400 21405 log.cc:824] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:26.504091 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: No bootstrap required, opened a new log
I20260929 19:02:26.505918 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 19:02:26.506072 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:26.506100 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b9cadaa3914aef9d41bcd158b7fc2e, State: Initialized, Role: FOLLOWER
I20260929 19:02:26.506228 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 19:02:26.506292 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:26.506318 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:26.506350 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:26.506953 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 19:02:26.507063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21337
I20260929 19:02:26.507089 21405 leader_election.cc:302] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14b9cadaa3914aef9d41bcd158b7fc2e; no voters:
I20260929 19:02:26.507195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:26.507316 21405 leader_election.cc:288] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:26.507390 21408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:26.507501 392 external_mini_cluster.cc:1428] Setting key 58e803681bf410f94ed910b5c1be5963
I20260929 19:02:26.507561 21408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [term 1 LEADER]: Becoming Leader. State: Replica: 14b9cadaa3914aef9d41bcd158b7fc2e, State: Running, Role: LEADER
I20260929 19:02:26.507612 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:26.507735 21408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 19:02:26.508613 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b9cadaa3914aef9d41bcd158b7fc2e. Latest consensus state: current_term: 1 leader_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260929 19:02:26.508778 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9cadaa3914aef9d41bcd158b7fc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260929 19:02:26.508816 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:26.509130 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:26.510141 21425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:26.510196 21425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:26.510243 21420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:26.510416 21420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:26.511905 21420 catalog_manager.cc:1347] Generated new cluster ID: 1e02d80e046147d381ccbce02e79f1ea
I20260929 19:02:26.511956 21420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:26.514423 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:26.508022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:50757 (local address 127.0.0.1:38629)
0929 19:02:26.508400 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:26.508410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:26.508447 (+ 37us) server_negotiation.cc:407] Connection header received
0929 19:02:26.508566 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:26.508573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:26.508631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:26.508801 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:26.509198 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:26.510709 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:26.512060 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:26.512657 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:26.512757 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:26.513599 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:26.513603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:26.513603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:26.513850 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:26.513885 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:26.513899 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:26.513979 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:26.514180 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:02:26.514207 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260929 19:02:26.515666 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:26.521381 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:26.517338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42959 (local address 127.0.0.1:38629)
0929 19:02:26.517370 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:26.517371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:26.517399 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:02:26.517498 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:26.517499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:26.517507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:26.517557 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:26.517732 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:26.518770 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:26.519946 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:26.520260 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:26.520329 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:26.521098 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:26.521100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:26.521100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:26.521104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:26.521117 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:26.521122 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:26.521176 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:26.521293 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 19:02:26.521305 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:02:26.548357 21420 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:26.548939 21420 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:26.552438 21420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14b9cadaa3914aef9d41bcd158b7fc2e: Generated new TSK 0
I20260929 19:02:26.552627 21420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:26.556743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21337
W20260929 19:02:26.561486 21322 connection.cc:535] client connection to 127.0.0.1:38629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38629): Connection reset by peer (error 104)
2026-09-29T19:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:26 dist-test-slave-gfsd krb5kdc[21436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:26 dist-test-slave-gfsd krb5kdc[21436](info): set up 2 sockets
Sep 29 19:02:26 dist-test-slave-gfsd krb5kdc[21436](info): commencing operation
krb5kdc: starting...
W20260929 19:02:27.607482 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:27 dist-test-slave-gfsd krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:27.726384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:27.807540 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:27.807797 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:27.807837 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:27.808985 21449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:27.809094 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:27.809121 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:27.809150 21449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:27.809180 21449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:27.810693 21449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:27.810973 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:27.811197 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:27.816424 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:27.816406 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:27.816437 21449 server_base.cc:1034] running on GCE node
W20260929 19:02:27.816407 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:27.817040 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:27.817392 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:27.818585 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790708547818573 us; error 50 us; skew 500 ppm
Sep 29 19:02:27 dist-test-slave-gfsd krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:27.821291 21449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:27.821756 21449 webserver.cc:460] Webserver started at http://127.0.0.1:34177/ using document root <none> and password file <none>
I20260929 19:02:27.821957 21449 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:27.821997 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:27.822093 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:27.823086 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "00e46a88662049e7b916e0744eaa813f"
format_stamp: "Formatted at 2026-09-29 19:02:27 on dist-test-slave-gfsd"
server_key: "4831b73f58578a3a17de080e1bb04fa6"
server_key_iv: "9496a60d5b2163d96948f0486d17fb89"
server_key_version: "encryptionkey@0"
I20260929 19:02:27.823470 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "00e46a88662049e7b916e0744eaa813f"
format_stamp: "Formatted at 2026-09-29 19:02:27 on dist-test-slave-gfsd"
server_key: "4831b73f58578a3a17de080e1bb04fa6"
server_key_iv: "9496a60d5b2163d96948f0486d17fb89"
server_key_version: "encryptionkey@0"
I20260929 19:02:27.824748 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:27.825537 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:27.825732 21449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:27.825816 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "00e46a88662049e7b916e0744eaa813f"
format_stamp: "Formatted at 2026-09-29 19:02:27 on dist-test-slave-gfsd"
server_key: "4831b73f58578a3a17de080e1bb04fa6"
server_key_iv: "9496a60d5b2163d96948f0486d17fb89"
server_key_version: "encryptionkey@0"
I20260929 19:02:27.825863 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:27.829089 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:27.831003 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:27.831105 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:27.834678 21449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33625
I20260929 19:02:27.834729 21516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33625 every 8 connection(s)
I20260929 19:02:27.835031 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:27.836102 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:27.838342 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: Bootstrap starting.
I20260929 19:02:27.838932 21517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:27.839221 21517 log.cc:824] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:27.839949 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: No bootstrap required, opened a new log
I20260929 19:02:27.841153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21449
I20260929 19:02:27.841282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:27.841502 392 external_mini_cluster.cc:1428] Setting key 621b9d15727da0103df42224319a658c
I20260929 19:02:27.841698 21517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } }
I20260929 19:02:27.841882 21517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:27.841972 21517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e46a88662049e7b916e0744eaa813f, State: Initialized, Role: FOLLOWER
I20260929 19:02:27.842394 21517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } }
I20260929 19:02:27.842486 21517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:27.842523 21517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:27.842579 21517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:27.843349 21517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } }
I20260929 19:02:27.843458 21517 leader_election.cc:302] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e46a88662049e7b916e0744eaa813f; no voters:
I20260929 19:02:27.843660 21517 leader_election.cc:288] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:27.843700 21522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:27.843807 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:27.842080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:47989 (local address 127.0.0.1:33625)
0929 19:02:27.842263 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:27.842272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:27.842308 (+ 36us) server_negotiation.cc:407] Connection header received
0929 19:02:27.842422 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:27.842429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:27.842458 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:27.842606 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:27.843232 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:27.843236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:27.843236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:27.843240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:27.843300 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:27.843317 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:27.843398 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:27.843518 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 19:02:27.843565 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260929 19:02:27.843925 21517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:27.845053 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:27.843988 21522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [term 1 LEADER]: Becoming Leader. State: Replica: 00e46a88662049e7b916e0744eaa813f, State: Running, Role: LEADER
I20260929 19:02:27.845433 21522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } }
I20260929 19:02:27.846170 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e46a88662049e7b916e0744eaa813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } } }
I20260929 19:02:27.846243 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:27.846328 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e46a88662049e7b916e0744eaa813f. Latest consensus state: current_term: 1 leader_uuid: "00e46a88662049e7b916e0744eaa813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e46a88662049e7b916e0744eaa813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33625 } } }
I20260929 19:02:27.846642 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:27.846963 21538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:27.847029 21538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:27.847141 21542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:27.847345 21542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:27.848645 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:27.846648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:43209 (local address 127.0.0.1:33625)
0929 19:02:27.846683 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:27.846686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:27.847433 (+ 747us) server_negotiation.cc:407] Connection header received
0929 19:02:27.847525 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:27.847526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:27.847532 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:27.847603 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:27.848398 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:27.848399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:27.848399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:27.848404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:27.848418 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:27.848422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:27.848486 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:27.848541 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 19:02:27.848556 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:27.848883 21542 catalog_manager.cc:1347] Generated new cluster ID: e8b383f8a4074cd29122182cf75b7a5c
I20260929 19:02:27.848930 21542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:27.865109 21542 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:27.865320 21542 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:27.868232 21542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e46a88662049e7b916e0744eaa813f: Generated new TSK 0
I20260929 19:02:27.868430 21542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:27.875619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21449
2026-09-29T19:02:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:27 dist-test-slave-gfsd krb5kdc[21548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:27 dist-test-slave-gfsd krb5kdc[21548](info): set up 2 sockets
Sep 29 19:02:27 dist-test-slave-gfsd krb5kdc[21548](info): commencing operation
krb5kdc: starting...
W20260929 19:02:28.919507 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:29.038568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:29.120884 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:29.121119 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:29.121156 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:29.122272 21561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:29.122339 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:29.122361 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:29.122383 21561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:29.122413 21561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:29.123934 21561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:29.124204 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:29.124426 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:29.129155 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:29.129329 21561 server_base.cc:1034] running on GCE node
W20260929 19:02:29.129163 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:29.129184 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:29.129643 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:29.129927 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:29.131100 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790708549131091 us; error 43 us; skew 500 ppm
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:29.137773 21561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:29.138196 21561 webserver.cc:460] Webserver started at http://127.0.0.1:36617/ using document root <none> and password file <none>
I20260929 19:02:29.138458 21561 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:29.138530 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:29.138675 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:29.139850 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "5949bda2061a4b798fc07fdc9a5b6faa"
format_stamp: "Formatted at 2026-09-29 19:02:29 on dist-test-slave-gfsd"
server_key: "92e55e303390cfca2785248e9034f015"
server_key_iv: "b5d201a511da271fb6cf60ce9de1e3df"
server_key_version: "encryptionkey@0"
I20260929 19:02:29.140218 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "5949bda2061a4b798fc07fdc9a5b6faa"
format_stamp: "Formatted at 2026-09-29 19:02:29 on dist-test-slave-gfsd"
server_key: "92e55e303390cfca2785248e9034f015"
server_key_iv: "b5d201a511da271fb6cf60ce9de1e3df"
server_key_version: "encryptionkey@0"
I20260929 19:02:29.141518 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:29.142333 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:29.142539 21561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:29.142644 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "5949bda2061a4b798fc07fdc9a5b6faa"
format_stamp: "Formatted at 2026-09-29 19:02:29 on dist-test-slave-gfsd"
server_key: "92e55e303390cfca2785248e9034f015"
server_key_iv: "b5d201a511da271fb6cf60ce9de1e3df"
server_key_version: "encryptionkey@0"
I20260929 19:02:29.142748 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:29.156490 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:29.158387 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:29.158551 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:29.162503 21561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39051
I20260929 19:02:29.162542 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39051 every 8 connection(s)
I20260929 19:02:29.163002 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:29.163942 21629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:29.166177 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa: Bootstrap starting.
I20260929 19:02:29.166751 21629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:29.167037 21629 log.cc:824] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:29.167891 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa: No bootstrap required, opened a new log
I20260929 19:02:29.169492 21629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } }
I20260929 19:02:29.169627 21629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:29.169649 21629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5949bda2061a4b798fc07fdc9a5b6faa, State: Initialized, Role: FOLLOWER
I20260929 19:02:29.169814 21629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } }
I20260929 19:02:29.169879 21629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:29.169900 21629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:29.169927 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:29.170450 21629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } }
I20260929 19:02:29.170574 21629 leader_election.cc:302] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5949bda2061a4b798fc07fdc9a5b6faa; no voters:
I20260929 19:02:29.170759 21629 leader_election.cc:288] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:29.170867 21632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:29.171051 21632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [term 1 LEADER]: Becoming Leader. State: Replica: 5949bda2061a4b798fc07fdc9a5b6faa, State: Running, Role: LEADER
I20260929 19:02:29.171044 21629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:29.171247 21632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } }
I20260929 19:02:29.171715 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5949bda2061a4b798fc07fdc9a5b6faa. Latest consensus state: current_term: 1 leader_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } } }
I20260929 19:02:29.171671 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5949bda2061a4b798fc07fdc9a5b6faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39051 } } }
I20260929 19:02:29.171854 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:29.171926 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:29.172317 21637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:29.172853 21637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:29.173184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21561
I20260929 19:02:29.173305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:29.173514 392 external_mini_cluster.cc:1428] Setting key b8cf741a19bae5e00daf0ea4ba1eda3f
I20260929 19:02:29.174436 21637 catalog_manager.cc:1347] Generated new cluster ID: 680a4fdbe82540529e39f834b56328a4
I20260929 19:02:29.174501 21637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:02:29.185848 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:29.174148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:39365 (local address 127.0.0.1:39051)
0929 19:02:29.174516 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:29.174524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:29.174558 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:02:29.174669 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:29.174676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:29.174734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:29.174880 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:29.175620 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:29.177116 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:29.178169 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:29.178811 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:29.178936 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:29.181796 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:29.181800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:29.181801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:29.181813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:29.184021 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:29.184436 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:29.184439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:29.184439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:29.184553 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:29.184782 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:29.184783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:29.184783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:29.185099 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:29.185271 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:29.185386 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 19:02:29.185603 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":63,"threads_started":1}
I20260929 19:02:29.186337 21637 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:29.186929 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:29.187065 21637 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:29.192355 21637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5949bda2061a4b798fc07fdc9a5b6faa: Generated new TSK 0
I20260929 19:02:29.192541 21637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:29.195330 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:29.188413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:47735 (local address 127.0.0.1:39051)
0929 19:02:29.188451 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:29.188453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:29.188586 (+ 133us) server_negotiation.cc:407] Connection header received
0929 19:02:29.188632 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:29.188633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:29.188642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:29.188699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:29.189300 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:29.190053 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:29.191117 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:29.191471 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:29.191536 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:29.192924 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:29.192925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:29.192925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:29.192930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:29.194209 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:29.194594 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:29.194595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:29.194595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:29.194644 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:29.194861 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:29.194862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:29.194862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:29.194952 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:29.195027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:29.195172 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:02:29.195218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:29.197621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21561
W20260929 19:02:29.202136 21546 connection.cc:535] client connection to 127.0.0.1:39051 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39051): Connection reset by peer (error 104)
2026-09-29T19:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21660](info): set up 2 sockets
Sep 29 19:02:29 dist-test-slave-gfsd krb5kdc[21660](info): commencing operation
krb5kdc: starting...
W20260929 19:02:30.227618 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:30 dist-test-slave-gfsd krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:30.350118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:30.430335 21673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:30.430589 21673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:30.430631 21673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:30.431818 21673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:30.431882 21673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:30.431907 21673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:30.431942 21673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:30.431972 21673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:30.433444 21673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:30.433729 21673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:30.433959 21673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:30.439038 21679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:30.439148 21678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:30.439041 21681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:30.439376 21673 server_base.cc:1034] running on GCE node
I20260929 19:02:30.439611 21673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:30.439862 21673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:30.441016 21673 hybrid_clock.cc:648] HybridClock initialized: now 1790708550441001 us; error 30 us; skew 500 ppm
Sep 29 19:02:30 dist-test-slave-gfsd krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:30.443610 21673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:30.444144 21673 webserver.cc:460] Webserver started at http://127.0.0.1:34161/ using document root <none> and password file <none>
I20260929 19:02:30.444413 21673 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:30.444465 21673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:30.444643 21673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:30.445788 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "d2fee4880f5f467986ae4d74fa87b052"
format_stamp: "Formatted at 2026-09-29 19:02:30 on dist-test-slave-gfsd"
server_key: "9b8570896f2bc42b86d04e1cee0d9b10"
server_key_iv: "b1d2608523d71d4a8f9b62560ff5d50b"
server_key_version: "encryptionkey@0"
I20260929 19:02:30.446161 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2fee4880f5f467986ae4d74fa87b052"
format_stamp: "Formatted at 2026-09-29 19:02:30 on dist-test-slave-gfsd"
server_key: "9b8570896f2bc42b86d04e1cee0d9b10"
server_key_iv: "b1d2608523d71d4a8f9b62560ff5d50b"
server_key_version: "encryptionkey@0"
I20260929 19:02:30.447618 21673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 19:02:30.448366 21688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:30.448546 21673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:30.448654 21673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "d2fee4880f5f467986ae4d74fa87b052"
format_stamp: "Formatted at 2026-09-29 19:02:30 on dist-test-slave-gfsd"
server_key: "9b8570896f2bc42b86d04e1cee0d9b10"
server_key_iv: "b1d2608523d71d4a8f9b62560ff5d50b"
server_key_version: "encryptionkey@0"
I20260929 19:02:30.448809 21673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:30.461088 21673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:30.462953 21673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:30.463109 21673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:30.467047 21673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43739
I20260929 19:02:30.467058 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43739 every 8 connection(s)
I20260929 19:02:30.467479 21673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:30.468451 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:30.470603 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052: Bootstrap starting.
I20260929 19:02:30.471165 21741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:30.471468 21741 log.cc:824] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:30.472244 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052: No bootstrap required, opened a new log
I20260929 19:02:30.473757 21741 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260929 19:02:30.473909 21741 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:30.473932 21741 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fee4880f5f467986ae4d74fa87b052, State: Initialized, Role: FOLLOWER
I20260929 19:02:30.474083 21741 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260929 19:02:30.474166 21741 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:30.474208 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:30.474277 21741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:30.474815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21673
I20260929 19:02:30.474917 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260929 19:02:30.474951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:30.475028 21741 leader_election.cc:302] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fee4880f5f467986ae4d74fa87b052; no voters:
I20260929 19:02:30.475173 392 external_mini_cluster.cc:1428] Setting key b1af5aa34501ee01acfa6436c427b13a
I20260929 19:02:30.475194 21741 leader_election.cc:288] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:30.475251 21744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:30.475492 21744 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [term 1 LEADER]: Becoming Leader. State: Replica: d2fee4880f5f467986ae4d74fa87b052, State: Running, Role: LEADER
I20260929 19:02:30.475643 21744 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260929 19:02:30.475474 21741 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:30.476022 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fee4880f5f467986ae4d74fa87b052. Latest consensus state: current_term: 1 leader_uuid: "d2fee4880f5f467986ae4d74fa87b052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } } }
I20260929 19:02:30.476100 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:30.476152 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fee4880f5f467986ae4d74fa87b052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fee4880f5f467986ae4d74fa87b052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } } }
I20260929 19:02:30.476210 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:30.476615 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:30.477507 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:30.480535 21750 catalog_manager.cc:1347] Generated new cluster ID: c652f0b6088947aebe4ebe08eef38c6b
I20260929 19:02:30.480635 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:30.482594 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:30.475718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:50577 (local address 127.0.0.1:43739)
0929 19:02:30.475938 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:30.475947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:30.475979 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:02:30.476098 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:30.476105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:30.476163 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:30.476359 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:30.476624 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:30.478373 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:30.479217 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:30.479865 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:30.479989 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:30.481841 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:30.481845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:30.481845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:30.482089 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:30.482127 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:30.482141 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:30.482229 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:30.482363 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 19:02:30.482389 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260929 19:02:30.483559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:30.494906 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:30.485049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:44961 (local address 127.0.0.1:43739)
0929 19:02:30.485085 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:30.485087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:30.491479 (+ 6392us) server_negotiation.cc:407] Connection header received
0929 19:02:30.491532 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:30.491533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:30.491544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:30.491606 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:30.491766 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:30.492504 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:30.493531 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:30.493869 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:30.493943 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:30.494651 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:30.494652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:30.494652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:30.494656 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:30.494670 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:30.494674 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:30.494728 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:30.494820 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 19:02:30.494832 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:30.511018 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:30.511646 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:30.516857 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fee4880f5f467986ae4d74fa87b052: Generated new TSK 0
I20260929 19:02:30.517037 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:30.522042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21673
W20260929 19:02:30.526420 21658 connection.cc:535] client connection to 127.0.0.1:43739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43739): Connection reset by peer (error 104)
2026-09-29T19:02:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:30 dist-test-slave-gfsd krb5kdc[21772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:30 dist-test-slave-gfsd krb5kdc[21772](info): set up 2 sockets
Sep 29 19:02:30 dist-test-slave-gfsd krb5kdc[21772](info): commencing operation
krb5kdc: starting...
W20260929 19:02:31.551591 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:31 dist-test-slave-gfsd krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:02:31.671942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:31.753007 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:31.753206 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:31.753227 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:31.754364 21785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:31.754405 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:31.754419 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:31.754431 21785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:31.754443 21785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:31.756124 21785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:31.756366 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:31.756584 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:31.761416 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:31.761555 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:31.761538 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:31.761538 21785 server_base.cc:1034] running on GCE node
I20260929 19:02:31.761901 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:31.762179 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:31.763352 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790708551763345 us; error 33 us; skew 500 ppm
Sep 29 19:02:31 dist-test-slave-gfsd krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:31.765805 21785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:02:31.766284 21785 webserver.cc:460] Webserver started at http://127.0.0.1:37781/ using document root <none> and password file <none>
I20260929 19:02:31.766530 21785 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:31.766593 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:31.766736 21785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:31.767877 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "d01b3acc3d1a444bb7f3d59a1b987840"
format_stamp: "Formatted at 2026-09-29 19:02:31 on dist-test-slave-gfsd"
server_key: "d56bbedf26ffc61f732cda80665d9143"
server_key_iv: "7fb902672f4bdefc07f51291c5ad2660"
server_key_version: "encryptionkey@0"
I20260929 19:02:31.768224 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "d01b3acc3d1a444bb7f3d59a1b987840"
format_stamp: "Formatted at 2026-09-29 19:02:31 on dist-test-slave-gfsd"
server_key: "d56bbedf26ffc61f732cda80665d9143"
server_key_iv: "7fb902672f4bdefc07f51291c5ad2660"
server_key_version: "encryptionkey@0"
I20260929 19:02:31.769510 21785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:02:31.770292 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:31.770480 21785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:31.770583 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "d01b3acc3d1a444bb7f3d59a1b987840"
format_stamp: "Formatted at 2026-09-29 19:02:31 on dist-test-slave-gfsd"
server_key: "d56bbedf26ffc61f732cda80665d9143"
server_key_iv: "7fb902672f4bdefc07f51291c5ad2660"
server_key_version: "encryptionkey@0"
I20260929 19:02:31.770658 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:31.774032 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:31.776401 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:31.776551 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:31.780629 21785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43969
I20260929 19:02:31.780665 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43969 every 8 connection(s)
I20260929 19:02:31.781018 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:31.781980 21853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:31.784421 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840: Bootstrap starting.
I20260929 19:02:31.784960 21853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:31.785241 21853 log.cc:824] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:31.785897 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840: No bootstrap required, opened a new log
I20260929 19:02:31.786345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21785
I20260929 19:02:31.786438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:31.786626 392 external_mini_cluster.cc:1428] Setting key ff4194f50cd5ec355906f0aa4c77bb69
I20260929 19:02:31.788388 21853 raft_consensus.cc:348] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } }
I20260929 19:02:31.788570 21853 raft_consensus.cc:374] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:31.788621 21853 raft_consensus.cc:729] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01b3acc3d1a444bb7f3d59a1b987840, State: Initialized, Role: FOLLOWER
I20260929 19:02:31.788760 21853 consensus_queue.cc:260] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } }
I20260929 19:02:31.788869 21853 raft_consensus.cc:388] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:31.788911 21853 raft_consensus.cc:482] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:31.789012 21853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:31.789247 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:31.787168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42013 (local address 127.0.0.1:43969)
0929 19:02:31.787374 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:31.787384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:31.787421 (+ 37us) server_negotiation.cc:407] Connection header received
0929 19:02:31.787912 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:31.787921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:31.787955 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:31.788140 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:31.788761 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:31.788764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:31.788765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:31.788770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:31.788818 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:31.788833 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:31.788918 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:31.788994 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 19:02:31.789038 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260929 19:02:31.790079 21853 raft_consensus.cc:504] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } }
I20260929 19:02:31.790133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:31.790201 21853 leader_election.cc:302] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d01b3acc3d1a444bb7f3d59a1b987840; no voters:
I20260929 19:02:31.790367 21853 leader_election.cc:288] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:31.790396 21858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:31.790515 21858 raft_consensus.cc:686] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [term 1 LEADER]: Becoming Leader. State: Replica: d01b3acc3d1a444bb7f3d59a1b987840, State: Running, Role: LEADER
I20260929 19:02:31.790642 21853 sys_catalog.cc:564] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:31.790659 21858 consensus_queue.cc:237] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } }
I20260929 19:02:31.791100 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } } }
I20260929 19:02:31.791317 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:31.791105 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d01b3acc3d1a444bb7f3d59a1b987840. Latest consensus state: current_term: 1 leader_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01b3acc3d1a444bb7f3d59a1b987840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43969 } } }
I20260929 19:02:31.791458 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:31.791613 21871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:31.792228 21871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:31.793502 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:31.791711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:44915 (local address 127.0.0.1:43969)
0929 19:02:31.791744 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:31.791745 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:31.792427 (+ 682us) server_negotiation.cc:407] Connection header received
0929 19:02:31.792537 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:31.792538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:31.792544 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:31.792578 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:31.793277 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:31.793278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:31.793279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:31.793283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:31.793297 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:31.793302 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:31.793360 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:31.793430 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 19:02:31.793443 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:02:31.794005 21871 catalog_manager.cc:1347] Generated new cluster ID: 80362261475143c7ac6521fce31eafa0
I20260929 19:02:31.794075 21871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:31.819474 21871 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:31.819670 21871 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:31.826143 21871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d01b3acc3d1a444bb7f3d59a1b987840: Generated new TSK 0
I20260929 19:02:31.826357 21871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:31.831410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21785
2026-09-29T19:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:31 dist-test-slave-gfsd krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:31 dist-test-slave-gfsd krb5kdc[21884](info): set up 2 sockets
Sep 29 19:02:31 dist-test-slave-gfsd krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260929 19:02:32.867429 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:32 dist-test-slave-gfsd krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:32.998224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:33.079010 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:33.079265 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:33.079329 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:33.080504 21897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:33.080567 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:33.080593 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:33.080622 21897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:33.080647 21897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:33.082145 21897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:33.082412 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:33.082626 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:33.087946 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:33.087970 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:33.087947 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:33.087961 21897 server_base.cc:1034] running on GCE node
I20260929 19:02:33.088387 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:33.088634 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:33.089833 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790708553089808 us; error 41 us; skew 500 ppm
Sep 29 19:02:33 dist-test-slave-gfsd krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:33.092396 21897 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:33.092869 21897 webserver.cc:460] Webserver started at http://127.0.98.62:37457/ using document root <none> and password file <none>
I20260929 19:02:33.093096 21897 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:33.093139 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:33.093322 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:33.094374 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "6f1872f761214c2fa9e48f111c5a47b7"
format_stamp: "Formatted at 2026-09-29 19:02:33 on dist-test-slave-gfsd"
server_key: "6cd97e967f2d24eb9951c4661d13f5ba"
server_key_iv: "4ead82fd74edf74068f7edf969260bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:33.094748 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f1872f761214c2fa9e48f111c5a47b7"
format_stamp: "Formatted at 2026-09-29 19:02:33 on dist-test-slave-gfsd"
server_key: "6cd97e967f2d24eb9951c4661d13f5ba"
server_key_iv: "4ead82fd74edf74068f7edf969260bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:33.096175 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:33.097013 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:33.097208 21897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:33.097285 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "6f1872f761214c2fa9e48f111c5a47b7"
format_stamp: "Formatted at 2026-09-29 19:02:33 on dist-test-slave-gfsd"
server_key: "6cd97e967f2d24eb9951c4661d13f5ba"
server_key_iv: "4ead82fd74edf74068f7edf969260bf1"
server_key_version: "encryptionkey@0"
I20260929 19:02:33.097385 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:33.113473 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:33.115723 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:33.115880 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:33.119669 21897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38195
I20260929 19:02:33.119719 21964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38195 every 8 connection(s)
I20260929 19:02:33.120060 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:33.120976 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:33.122752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 21897
I20260929 19:02:33.122843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:33.123051 392 external_mini_cluster.cc:1428] Setting key 46f354bc55070ec1b37bee4c3739df90
I20260929 19:02:33.123494 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7: Bootstrap starting.
I20260929 19:02:33.124281 21965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:33.124595 21965 log.cc:824] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:33.125264 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7: No bootstrap required, opened a new log
I20260929 19:02:33.127503 21965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } }
I20260929 19:02:33.127663 21965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:33.127696 21965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1872f761214c2fa9e48f111c5a47b7, State: Initialized, Role: FOLLOWER
I20260929 19:02:33.127815 21965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } }
I20260929 19:02:33.127894 21965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:33.127933 21965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:33.127990 21965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:33.128609 21965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } }
I20260929 19:02:33.128723 21965 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1872f761214c2fa9e48f111c5a47b7; no voters:
I20260929 19:02:33.128918 21965 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:33.128942 21970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:33.129055 21970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1872f761214c2fa9e48f111c5a47b7, State: Running, Role: LEADER
I20260929 19:02:33.129180 21965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:33.129186 21970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } }
I20260929 19:02:33.129575 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1872f761214c2fa9e48f111c5a47b7. Latest consensus state: current_term: 1 leader_uuid: "6f1872f761214c2fa9e48f111c5a47b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } } }
I20260929 19:02:33.129586 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1872f761214c2fa9e48f111c5a47b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1872f761214c2fa9e48f111c5a47b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38195 } } }
I20260929 19:02:33.129739 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:33.129827 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7 [sys.catalog]: This master's current role is: LEADER
Sep 29 19:02:33 dist-test-slave-gfsd krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:33.130335 21979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:33.131057 21979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:33.132666 21979 catalog_manager.cc:1347] Generated new cluster ID: 36fd3b17033741b9a2e70360faef41a8
I20260929 19:02:33.133098 21979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:33.134598 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:33.123562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60967 (local address 127.0.98.62:38195)
0929 19:02:33.123728 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:33.123737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:33.123768 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:02:33.124369 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:33.124390 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:33.124462 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:33.124617 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:33.125207 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:33.126571 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:33.127372 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:33.127981 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:33.128098 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:33.130941 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:33.130945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:33.130945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:33.130956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:33.132908 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:33.133334 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:33.133337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:33.133338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:33.133444 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:33.133677 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:33.133678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:33.133678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:33.133984 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:33.134142 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:33.134264 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 19:02:33.134376 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260929 19:02:33.135470 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:33.141906 21979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:33.142442 21979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:33.148216 21979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1872f761214c2fa9e48f111c5a47b7: Generated new TSK 0
I20260929 19:02:33.148384 21979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:33.149016 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:33.141718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50263 (local address 127.0.98.62:38195)
0929 19:02:33.141762 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:33.141764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:33.141901 (+ 137us) server_negotiation.cc:407] Connection header received
0929 19:02:33.142015 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:33.142016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:33.142051 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:33.142100 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:33.142745 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:33.143466 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:33.144481 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:33.144777 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:33.144842 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:33.146212 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:33.146213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:33.146214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:33.146220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:33.147788 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:33.148262 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:33.148263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:33.148263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:33.148323 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:33.148541 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:33.148541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:33.148542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:33.148664 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:33.148771 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:33.148857 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:02:33.148929 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:02:33.150835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 21897
W20260929 19:02:33.154898 21882 connection.cc:535] client connection to 127.0.98.62:38195 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38195): Connection reset by peer (error 104)
2026-09-29T19:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:33 dist-test-slave-gfsd krb5kdc[21996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:33 dist-test-slave-gfsd krb5kdc[21996](info): set up 2 sockets
Sep 29 19:02:33 dist-test-slave-gfsd krb5kdc[21996](info): commencing operation
krb5kdc: starting...
W20260929 19:02:34.191401 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:34 dist-test-slave-gfsd krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:34.311378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:34.390916 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:34.391111 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:34.391134 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:34.392282 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:34.392324 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:34.392338 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:34.392351 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:34.392400 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:34.393803 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:34.394071 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:34.394294 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:02:34.399140 22009 server_base.cc:1034] running on GCE node
W20260929 19:02:34.399119 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:34.399107 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:34.399121 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:34.399590 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:34.399919 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:34.401103 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790708554401082 us; error 46 us; skew 500 ppm
Sep 29 19:02:34 dist-test-slave-gfsd krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:34.407441 22009 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:34.407941 22009 webserver.cc:460] Webserver started at http://127.0.98.62:44571/ using document root <none> and password file <none>
I20260929 19:02:34.408161 22009 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:34.408224 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:34.408362 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:34.409386 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "2c2341cf2ab34342ae10a984c531c439"
format_stamp: "Formatted at 2026-09-29 19:02:34 on dist-test-slave-gfsd"
server_key: "5e94774f0e9297de4060453e0483d68e"
server_key_iv: "8ef00ee59ad2885ae7eef01a966b767f"
server_key_version: "encryptionkey@0"
I20260929 19:02:34.409746 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c2341cf2ab34342ae10a984c531c439"
format_stamp: "Formatted at 2026-09-29 19:02:34 on dist-test-slave-gfsd"
server_key: "5e94774f0e9297de4060453e0483d68e"
server_key_iv: "8ef00ee59ad2885ae7eef01a966b767f"
server_key_version: "encryptionkey@0"
I20260929 19:02:34.411037 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:34.411911 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:34.412108 22009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:02:34.412205 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "2c2341cf2ab34342ae10a984c531c439"
format_stamp: "Formatted at 2026-09-29 19:02:34 on dist-test-slave-gfsd"
server_key: "5e94774f0e9297de4060453e0483d68e"
server_key_iv: "8ef00ee59ad2885ae7eef01a966b767f"
server_key_version: "encryptionkey@0"
I20260929 19:02:34.412293 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:34.427475 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:34.429382 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:34.429548 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:34.433652 22009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45241
I20260929 19:02:34.433686 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45241 every 8 connection(s)
I20260929 19:02:34.434042 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:34.434882 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:34.435972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22009
I20260929 19:02:34.436091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:34.436307 392 external_mini_cluster.cc:1428] Setting key 74be5d6524b8bdf46a4a6f142ea9fca4
I20260929 19:02:34.437400 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439: Bootstrap starting.
I20260929 19:02:34.438079 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:34.438365 22077 log.cc:824] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:34.439025 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439: No bootstrap required, opened a new log
I20260929 19:02:34.441455 22077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260929 19:02:34.441584 22077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:34.441606 22077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2341cf2ab34342ae10a984c531c439, State: Initialized, Role: FOLLOWER
I20260929 19:02:34.441740 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260929 19:02:34.441830 22077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:34.441876 22077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:34.441931 22077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:34.442148 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:34.436866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42445 (local address 127.0.98.62:45241)
0929 19:02:34.437030 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:34.437037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:34.437060 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:02:34.437483 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:34.437493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:34.437551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:34.437663 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:34.437942 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:34.439036 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:34.440249 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:34.440635 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:34.440720 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:34.441455 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:34.441458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:34.441459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:34.441643 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:34.441683 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:34.441701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:34.441795 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:34.441944 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 19:02:34.441973 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260929 19:02:34.442560 22077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260929 19:02:34.442682 22077 leader_election.cc:302] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c2341cf2ab34342ae10a984c531c439; no voters:
I20260929 19:02:34.442854 22077 leader_election.cc:288] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:34.442898 22082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:34.443079 22082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [term 1 LEADER]: Becoming Leader. State: Replica: 2c2341cf2ab34342ae10a984c531c439, State: Running, Role: LEADER
I20260929 19:02:34.443136 22077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:34.443230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:34.443217 22082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260929 19:02:34.443540 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c2341cf2ab34342ae10a984c531c439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } } }
I20260929 19:02:34.443526 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c2341cf2ab34342ae10a984c531c439. Latest consensus state: current_term: 1 leader_uuid: "2c2341cf2ab34342ae10a984c531c439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2341cf2ab34342ae10a984c531c439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } } }
I20260929 19:02:34.443617 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:34.443625 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:34.444041 22088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:34.444581 22088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:34.447376 22088 catalog_manager.cc:1347] Generated new cluster ID: b84685df8ea44297b5a8380b8666c199
I20260929 19:02:34.447470 22088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:34.447984 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:34.444694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39399 (local address 127.0.98.62:45241)
0929 19:02:34.444725 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:34.444726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:34.444970 (+ 244us) server_negotiation.cc:407] Connection header received
0929 19:02:34.445044 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:34.445045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:34.445054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:34.445116 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:34.445276 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:34.445736 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:34.446676 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:34.446885 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:34.446970 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:34.447737 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:34.447738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:34.447739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:34.447743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:34.447757 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:34.447760 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:34.447811 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:34.447910 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:02:34.447919 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:02:34.463088 22088 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:34.463663 22088 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:34.468941 22088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c2341cf2ab34342ae10a984c531c439: Generated new TSK 0
I20260929 19:02:34.469122 22088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:34.473431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22009
W20260929 19:02:34.477574 21994 connection.cc:535] client connection to 127.0.98.62:45241 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45241): Connection reset by peer (error 104)
2026-09-29T19:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:34 dist-test-slave-gfsd krb5kdc[22108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:34 dist-test-slave-gfsd krb5kdc[22108](info): set up 2 sockets
Sep 29 19:02:34 dist-test-slave-gfsd krb5kdc[22108](info): commencing operation
krb5kdc: starting...
W20260929 19:02:35.511451 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:35 dist-test-slave-gfsd krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:35.631697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:35.711961 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:35.712153 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:35.712175 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:35.713217 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:35.713249 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:35.713263 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:35.713276 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:35.713289 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:35.714638 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:35.714824 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:35.714982 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:35.719585 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:35.719630 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:35.719749 22121 server_base.cc:1034] running on GCE node
W20260929 19:02:35.719648 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:35.720101 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:35.720387 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:35.721545 22121 hybrid_clock.cc:648] HybridClock initialized: now 1790708555721535 us; error 33 us; skew 500 ppm
Sep 29 19:02:35 dist-test-slave-gfsd krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:35.723975 22121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:35.724340 22121 webserver.cc:460] Webserver started at http://127.0.98.62:41627/ using document root <none> and password file <none>
I20260929 19:02:35.724522 22121 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:35.724561 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:35.724644 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:35.725595 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "a276ba53370b40c990d9c974b3e01ff8"
format_stamp: "Formatted at 2026-09-29 19:02:35 on dist-test-slave-gfsd"
server_key: "05b3f2d7134112e0ec3739eccb8b0bac"
server_key_iv: "3fda63c5a4ad0d220ef03e159b578430"
server_key_version: "encryptionkey@0"
I20260929 19:02:35.725965 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "a276ba53370b40c990d9c974b3e01ff8"
format_stamp: "Formatted at 2026-09-29 19:02:35 on dist-test-slave-gfsd"
server_key: "05b3f2d7134112e0ec3739eccb8b0bac"
server_key_iv: "3fda63c5a4ad0d220ef03e159b578430"
server_key_version: "encryptionkey@0"
I20260929 19:02:35.727258 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:35.728129 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:35.728348 22121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:35.728442 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "a276ba53370b40c990d9c974b3e01ff8"
format_stamp: "Formatted at 2026-09-29 19:02:35 on dist-test-slave-gfsd"
server_key: "05b3f2d7134112e0ec3739eccb8b0bac"
server_key_iv: "3fda63c5a4ad0d220ef03e159b578430"
server_key_version: "encryptionkey@0"
I20260929 19:02:35.728502 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:35.731957 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:35.733825 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:35.733937 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:35.737650 22121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34629
I20260929 19:02:35.737743 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34629 every 8 connection(s)
I20260929 19:02:35.738023 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:35.739017 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:35.741169 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8: Bootstrap starting.
I20260929 19:02:35.741711 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:35.741984 22189 log.cc:824] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:35.742663 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8: No bootstrap required, opened a new log
I20260929 19:02:35.744367 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260929 19:02:35.744515 22189 raft_consensus.cc:374] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:35.744560 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a276ba53370b40c990d9c974b3e01ff8, State: Initialized, Role: FOLLOWER
I20260929 19:02:35.744697 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260929 19:02:35.744776 22189 raft_consensus.cc:388] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:35.744818 22189 raft_consensus.cc:482] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:35.744864 22189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:35.745384 22189 raft_consensus.cc:504] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260929 19:02:35.745496 22189 leader_election.cc:302] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a276ba53370b40c990d9c974b3e01ff8; no voters:
I20260929 19:02:35.745663 22189 leader_election.cc:288] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:35.745704 22192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:35.745884 22192 raft_consensus.cc:686] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [term 1 LEADER]: Becoming Leader. State: Replica: a276ba53370b40c990d9c974b3e01ff8, State: Running, Role: LEADER
I20260929 19:02:35.745904 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:35.745970 22192 consensus_queue.cc:237] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260929 19:02:35.746230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22121
I20260929 19:02:35.746357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:35.746451 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a276ba53370b40c990d9c974b3e01ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260929 19:02:35.746541 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:35.746556 392 external_mini_cluster.cc:1428] Setting key 2f99d8fd396b38cac61d13c6e1a12186
I20260929 19:02:35.746696 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a276ba53370b40c990d9c974b3e01ff8. Latest consensus state: current_term: 1 leader_uuid: "a276ba53370b40c990d9c974b3e01ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a276ba53370b40c990d9c974b3e01ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260929 19:02:35.746796 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:35.747027 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:35.747628 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:35.749006 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:35.746984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52467 (local address 127.0.98.62:34629)
0929 19:02:35.747154 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:35.747163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:35.747519 (+ 356us) server_negotiation.cc:407] Connection header received
0929 19:02:35.747669 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:35.747702 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:35.747739 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:35.747907 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:35.748554 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:35.748557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:35.748558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:35.748562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:35.748606 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:35.748620 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:35.748691 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:35.748796 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:02:35.748836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260929 19:02:35.749661 22196 catalog_manager.cc:1347] Generated new cluster ID: 065672fd01c040ef9b089cb1485ac84a
I20260929 19:02:35.749722 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:35.749946 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:35.756374 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:35.751558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58957 (local address 127.0.98.62:34629)
0929 19:02:35.751589 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:35.751590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:35.755425 (+ 3835us) server_negotiation.cc:407] Connection header received
0929 19:02:35.755496 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:35.755497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:35.755503 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:35.755564 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:35.756147 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:35.756148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:35.756148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:35.756152 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:35.756168 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:35.756173 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:35.756244 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:35.756287 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 19:02:35.756299 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:02:35.767556 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:35.767728 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:35.776187 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a276ba53370b40c990d9c974b3e01ff8: Generated new TSK 0
I20260929 19:02:35.776424 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:35.784375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22121
2026-09-29T19:02:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:35 dist-test-slave-gfsd krb5kdc[22220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:35 dist-test-slave-gfsd krb5kdc[22220](info): set up 2 sockets
Sep 29 19:02:35 dist-test-slave-gfsd krb5kdc[22220](info): commencing operation
krb5kdc: starting...
W20260929 19:02:36.847602 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:36 dist-test-slave-gfsd krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:36.965734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:37.048127 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:37.048370 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:37.048409 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:37.049544 22233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:37.049610 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:37.049633 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:37.049655 22233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:37.049681 22233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:37.051132 22233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:37.051438 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:37.051657 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:37.056663 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:37.056658 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:37.056658 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:37.056923 22233 server_base.cc:1034] running on GCE node
I20260929 19:02:37.057183 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:37.057503 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:37.058676 22233 hybrid_clock.cc:648] HybridClock initialized: now 1790708557058665 us; error 37 us; skew 500 ppm
Sep 29 19:02:37 dist-test-slave-gfsd krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:37.061435 22233 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:37.061877 22233 webserver.cc:460] Webserver started at http://127.0.98.62:40969/ using document root <none> and password file <none>
I20260929 19:02:37.062099 22233 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:37.062165 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:37.062318 22233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:37.063485 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "bbe151fbe280471c88783c50b7d5eec8"
format_stamp: "Formatted at 2026-09-29 19:02:37 on dist-test-slave-gfsd"
server_key: "12ca16ba2c1f95a945633ce93a19c8df"
server_key_iv: "a6a88a38e4939bcf092405af8e6935f0"
server_key_version: "encryptionkey@0"
I20260929 19:02:37.063884 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbe151fbe280471c88783c50b7d5eec8"
format_stamp: "Formatted at 2026-09-29 19:02:37 on dist-test-slave-gfsd"
server_key: "12ca16ba2c1f95a945633ce93a19c8df"
server_key_iv: "a6a88a38e4939bcf092405af8e6935f0"
server_key_version: "encryptionkey@0"
I20260929 19:02:37.065186 22233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:37.065948 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:37.066135 22233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:37.066242 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "bbe151fbe280471c88783c50b7d5eec8"
format_stamp: "Formatted at 2026-09-29 19:02:37 on dist-test-slave-gfsd"
server_key: "12ca16ba2c1f95a945633ce93a19c8df"
server_key_iv: "a6a88a38e4939bcf092405af8e6935f0"
server_key_version: "encryptionkey@0"
I20260929 19:02:37.066324 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:37.114341 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:37.116588 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:37.116782 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:37.121037 22233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37801
I20260929 19:02:37.121075 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37801 every 8 connection(s)
I20260929 19:02:37.121461 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:37.122375 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:37.124784 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8: Bootstrap starting.
I20260929 19:02:37.125321 22301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:37.125545 22301 log.cc:824] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:37.126260 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8: No bootstrap required, opened a new log
I20260929 19:02:37.127923 22301 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 19:02:37.128065 22301 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:37.128086 22301 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151fbe280471c88783c50b7d5eec8, State: Initialized, Role: FOLLOWER
I20260929 19:02:37.128228 22301 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 19:02:37.128309 22301 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:37.128333 22301 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:37.128362 22301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:37.128830 22301 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 19:02:37.128912 22301 leader_election.cc:302] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe151fbe280471c88783c50b7d5eec8; no voters:
I20260929 19:02:37.129046 22301 leader_election.cc:288] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:37.129123 22304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:37.129314 22304 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [term 1 LEADER]: Becoming Leader. State: Replica: bbe151fbe280471c88783c50b7d5eec8, State: Running, Role: LEADER
I20260929 19:02:37.129413 22304 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 19:02:37.129323 22301 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:37.129890 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe151fbe280471c88783c50b7d5eec8. Latest consensus state: current_term: 1 leader_uuid: "bbe151fbe280471c88783c50b7d5eec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 19:02:37.129974 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:37.130020 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe151fbe280471c88783c50b7d5eec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe151fbe280471c88783c50b7d5eec8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 19:02:37.130105 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:37.130522 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:37.131100 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:37.131100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22233
I20260929 19:02:37.131208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:37.131459 392 external_mini_cluster.cc:1428] Setting key 38e03c900635bf836f4916c31033e2f5
I20260929 19:02:37.132893 22312 catalog_manager.cc:1347] Generated new cluster ID: 47d081fa4c5947e1987c9f2c07c14956
I20260929 19:02:37.132941 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:37 dist-test-slave-gfsd krb5kdc[22220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:37.146293 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:37.132042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38237 (local address 127.0.98.62:37801)
0929 19:02:37.132235 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:37.132244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:37.132275 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:02:37.132408 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:37.132416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:37.132490 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:37.132647 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:37.135954 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:37.137916 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:37.138998 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:37.139693 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:37.139815 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:37.142641 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:37.142646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:37.142646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:37.142659 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:37.144611 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:37.145025 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:37.145028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:37.145028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:37.145136 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:37.145348 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:37.145348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:37.145349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:37.145651 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:37.145789 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:37.145916 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 19:02:37.146047 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260929 19:02:37.147199 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:37.156836 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:37.157361 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:37.163362 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:37.155627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58093 (local address 127.0.98.62:37801)
0929 19:02:37.155690 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:37.155692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:37.155810 (+ 118us) server_negotiation.cc:407] Connection header received
0929 19:02:37.155867 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:37.155868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:37.155878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:37.155949 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:37.156681 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:37.157453 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:37.158438 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:37.158753 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:37.158825 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:37.160329 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:37.160330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:37.160331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:37.160338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:37.161965 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:37.162334 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:37.162335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:37.162335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:37.162395 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:37.162590 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:37.162592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:37.162592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:37.162721 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:37.162819 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:37.162984 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:02:37.163081 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:02:37.163542 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe151fbe280471c88783c50b7d5eec8: Generated new TSK 0
I20260929 19:02:37.163734 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:37.168944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22233
2026-09-29T19:02:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:37 dist-test-slave-gfsd krb5kdc[22332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:37 dist-test-slave-gfsd krb5kdc[22332](info): set up 2 sockets
Sep 29 19:02:37 dist-test-slave-gfsd krb5kdc[22332](info): commencing operation
krb5kdc: starting...
W20260929 19:02:38.247474 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:38 dist-test-slave-gfsd krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:38.366209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38931
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:38.453027 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:38.453238 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:38.453261 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:38.454420 22345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:38.454464 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:38.454479 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:38.454491 22345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:38.454504 22345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:38.456015 22345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:38.456251 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:38.456488 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:38.461413 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:38.461428 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:38.461434 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:38.461743 22345 server_base.cc:1034] running on GCE node
I20260929 19:02:38.461987 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:38.462299 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:38.463455 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790708558463439 us; error 38 us; skew 500 ppm
Sep 29 19:02:38 dist-test-slave-gfsd krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:38.465997 22345 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:38.466362 22345 webserver.cc:460] Webserver started at http://127.0.98.62:43499/ using document root <none> and password file <none>
I20260929 19:02:38.466536 22345 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:38.466575 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:38.466658 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:38.467731 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "b5c52b0b33af4928933dbc855a195aae"
format_stamp: "Formatted at 2026-09-29 19:02:38 on dist-test-slave-gfsd"
server_key: "22d005e4c7601ce768e4be8f904e3463"
server_key_iv: "515896fe1c690d866fa846dca08bf162"
server_key_version: "encryptionkey@0"
I20260929 19:02:38.468035 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5c52b0b33af4928933dbc855a195aae"
format_stamp: "Formatted at 2026-09-29 19:02:38 on dist-test-slave-gfsd"
server_key: "22d005e4c7601ce768e4be8f904e3463"
server_key_iv: "515896fe1c690d866fa846dca08bf162"
server_key_version: "encryptionkey@0"
I20260929 19:02:38.469287 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:38.469986 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:38.470151 22345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:38.470258 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "b5c52b0b33af4928933dbc855a195aae"
format_stamp: "Formatted at 2026-09-29 19:02:38 on dist-test-slave-gfsd"
server_key: "22d005e4c7601ce768e4be8f904e3463"
server_key_iv: "515896fe1c690d866fa846dca08bf162"
server_key_version: "encryptionkey@0"
I20260929 19:02:38.470335 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:38.484913 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:38.486850 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:38.487010 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:38.491390 22345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38931
I20260929 19:02:38.491436 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38931 every 8 connection(s)
I20260929 19:02:38.491802 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:38.492628 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:38.494889 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae: Bootstrap starting.
I20260929 19:02:38.495540 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:38.495827 22413 log.cc:824] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:38.496543 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae: No bootstrap required, opened a new log
I20260929 19:02:38.498267 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260929 19:02:38.498402 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:38.498425 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c52b0b33af4928933dbc855a195aae, State: Initialized, Role: FOLLOWER
I20260929 19:02:38.498602 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260929 19:02:38.498668 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:38.498698 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:38.498759 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:38.499325 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260929 19:02:38.499430 22413 leader_election.cc:302] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c52b0b33af4928933dbc855a195aae; no voters:
I20260929 19:02:38.499603 22413 leader_election.cc:288] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:38.499626 22416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:38.499799 22416 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [term 1 LEADER]: Becoming Leader. State: Replica: b5c52b0b33af4928933dbc855a195aae, State: Running, Role: LEADER
I20260929 19:02:38.499852 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:38.499959 22416 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260929 19:02:38.500321 22417 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5c52b0b33af4928933dbc855a195aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } } }
I20260929 19:02:38.500417 22417 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:38.500326 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5c52b0b33af4928933dbc855a195aae. Latest consensus state: current_term: 1 leader_uuid: "b5c52b0b33af4928933dbc855a195aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c52b0b33af4928933dbc855a195aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } } }
I20260929 19:02:38.500501 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:38.500658 22420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:38.500957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22345
I20260929 19:02:38.501084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:38.501248 392 external_mini_cluster.cc:1428] Setting key 08fa2fceed4a36cd42ce94a5ba641e49
I20260929 19:02:38.502100 22420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:38.503667 22420 catalog_manager.cc:1347] Generated new cluster ID: d5bbd3ef77bd4fb4bca59e3acacdb5ba
I20260929 19:02:38.503727 22420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:38.508430 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:38.501745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58111 (local address 127.0.98.62:38931)
0929 19:02:38.501917 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:38.501927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:38.502913 (+ 986us) server_negotiation.cc:407] Connection header received
0929 19:02:38.503180 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:38.503187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:38.503244 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:38.503424 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:38.503759 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:38.505140 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:38.506212 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:38.506809 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:38.506892 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:38.507739 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:38.507742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:38.507742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:38.507939 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:38.507979 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:38.507994 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:38.508075 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:38.508143 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 19:02:38.508219 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260929 19:02:38.509407 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:38.512948 22420 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:38.513486 22420 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:38.516146 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:38.512615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41747 (local address 127.0.98.62:38931)
0929 19:02:38.512649 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:38.512651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:38.512794 (+ 143us) server_negotiation.cc:407] Connection header received
0929 19:02:38.512893 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:38.512894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:38.512904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:38.512958 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:38.513086 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:38.513876 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:38.514844 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:38.515155 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:38.515232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:38.515895 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:38.515897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:38.515897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:38.515901 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:38.515916 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:38.515920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:38.515972 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:38.516035 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 19:02:38.516075 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:38.523458 22420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5c52b0b33af4928933dbc855a195aae: Generated new TSK 0
I20260929 19:02:38.523643 22420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:38.532079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22345
2026-09-29T19:02:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:38 dist-test-slave-gfsd krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:38 dist-test-slave-gfsd krb5kdc[22444](info): set up 2 sockets
Sep 29 19:02:38 dist-test-slave-gfsd krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260929 19:02:39.575742 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:39 dist-test-slave-gfsd krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:39.701658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:39.782398 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:39.782601 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:39.782624 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:39.783788 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:39.783830 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:39.783844 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:39.783857 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:39.783869 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:39.785368 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:39.785611 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:39.785862 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:39.790776 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:39.790827 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:39.790835 22457 server_base.cc:1034] running on GCE node
W20260929 19:02:39.790768 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:39.791232 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:39.791535 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:39.792697 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790708559792684 us; error 42 us; skew 500 ppm
Sep 29 19:02:39 dist-test-slave-gfsd krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:39.799328 22457 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:39.799825 22457 webserver.cc:460] Webserver started at http://127.0.98.62:33073/ using document root <none> and password file <none>
I20260929 19:02:39.800066 22457 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:39.800158 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:39.800307 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:39.801393 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "5d4c878ffe6641b8b7b57f08c8afda68"
format_stamp: "Formatted at 2026-09-29 19:02:39 on dist-test-slave-gfsd"
server_key: "0444a25c74ffb20d011170f723756002"
server_key_iv: "0fa12fd9ef2a68e9a59b790a531473bc"
server_key_version: "encryptionkey@0"
I20260929 19:02:39.801774 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d4c878ffe6641b8b7b57f08c8afda68"
format_stamp: "Formatted at 2026-09-29 19:02:39 on dist-test-slave-gfsd"
server_key: "0444a25c74ffb20d011170f723756002"
server_key_iv: "0fa12fd9ef2a68e9a59b790a531473bc"
server_key_version: "encryptionkey@0"
I20260929 19:02:39.803193 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:39.804041 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:39.804245 22457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:39.804355 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "5d4c878ffe6641b8b7b57f08c8afda68"
format_stamp: "Formatted at 2026-09-29 19:02:39 on dist-test-slave-gfsd"
server_key: "0444a25c74ffb20d011170f723756002"
server_key_iv: "0fa12fd9ef2a68e9a59b790a531473bc"
server_key_version: "encryptionkey@0"
I20260929 19:02:39.804430 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:39.807847 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:39.810026 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:39.810221 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:39.814025 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34685
I20260929 19:02:39.814074 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34685 every 8 connection(s)
I20260929 19:02:39.814419 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:39.815274 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:39.816354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22457
I20260929 19:02:39.816485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:39.816701 392 external_mini_cluster.cc:1428] Setting key 2e6e88765ed598272b3b5add095f4a28
I20260929 19:02:39.817935 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68: Bootstrap starting.
I20260929 19:02:39.818650 22525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:39.818988 22525 log.cc:824] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:39.819412 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:39.817243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41277 (local address 127.0.98.62:34685)
0929 19:02:39.817431 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:39.817440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:39.817472 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:02:39.817898 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:39.817904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:39.817934 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:39.818097 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:39.818836 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:39.818839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:39.818839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:39.818842 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:39.818879 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:39.818888 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:39.818958 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:39.819037 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:02:39.819217 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260929 19:02:39.819752 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68: No bootstrap required, opened a new log
I20260929 19:02:39.820446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:39.822283 22525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260929 19:02:39.822459 22525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:39.822487 22525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c878ffe6641b8b7b57f08c8afda68, State: Initialized, Role: FOLLOWER
I20260929 19:02:39.822625 22525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260929 19:02:39.822736 22525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:39.822784 22525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:39.822844 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:39.823165 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:39.822060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44913 (local address 127.0.98.62:34685)
0929 19:02:39.822101 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:39.822103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:39.822219 (+ 116us) server_negotiation.cc:407] Connection header received
0929 19:02:39.822263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:39.822263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:39.822268 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:39.822339 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:39.822938 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:39.822939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:39.822939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:02:39.822944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:39.822958 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:02:39.822962 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:39.823019 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:39.823086 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:02:39.823102 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:02:39.823491 22525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260929 19:02:39.823613 22525 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4c878ffe6641b8b7b57f08c8afda68; no voters:
I20260929 19:02:39.823774 22525 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:39.823800 22535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:39.823909 22535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4c878ffe6641b8b7b57f08c8afda68, State: Running, Role: LEADER
I20260929 19:02:39.824031 22525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:39.824014 22535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260929 19:02:39.824440 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } } }
I20260929 19:02:39.824451 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4c878ffe6641b8b7b57f08c8afda68. Latest consensus state: current_term: 1 leader_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c878ffe6641b8b7b57f08c8afda68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } } }
I20260929 19:02:39.824602 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:39.824630 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:39.825018 22544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:39.825570 22544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:39.827032 22544 catalog_manager.cc:1347] Generated new cluster ID: f45c7e69fa9e4d77a3de53af906c7622
I20260929 19:02:39.827083 22544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:39.838146 22544 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:39.838356 22544 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:39.848297 22544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d4c878ffe6641b8b7b57f08c8afda68: Generated new TSK 0
I20260929 19:02:39.848488 22544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:39.855496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22457
2026-09-29T19:02:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1321 ms)
[----------] 18 tests from AuthTokenIssuingTest (23931 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:39 dist-test-slave-gfsd krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:39 dist-test-slave-gfsd krb5kdc[22556](info): set up 2 sockets
Sep 29 19:02:39 dist-test-slave-gfsd krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260929 19:02:40.887454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:40 dist-test-slave-gfsd krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:41.007144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:41.091212 22569 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:41.091502 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:41.091526 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:41.091547 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:41.092689 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:41.092725 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:41.092739 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:41.092751 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:41.092763 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:41.094144 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:41.094329 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:41.094501 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:41.099320 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:41.099406 22569 server_base.cc:1034] running on GCE node
W20260929 19:02:41.099427 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:41.099426 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:41.099817 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:41.100098 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:41.101256 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790708561101249 us; error 35 us; skew 500 ppm
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:41.103863 22569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:41.104303 22569 webserver.cc:460] Webserver started at http://127.0.98.62:35861/ using document root <none> and password file <none>
I20260929 19:02:41.104481 22569 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:41.104521 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:41.104605 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:41.105580 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "4a4365e8496a4382894cabef1dc77ec3"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "648352255c9eb1b9dd2ce6861c8f4c78"
server_key_iv: "ed102fe4e660cb77973d9e44f8d6f9e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.105885 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a4365e8496a4382894cabef1dc77ec3"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "648352255c9eb1b9dd2ce6861c8f4c78"
server_key_iv: "ed102fe4e660cb77973d9e44f8d6f9e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.107170 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:41.108026 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:41.108242 22569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:41.108340 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "4a4365e8496a4382894cabef1dc77ec3"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "648352255c9eb1b9dd2ce6861c8f4c78"
server_key_iv: "ed102fe4e660cb77973d9e44f8d6f9e7"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.108415 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:41.121491 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:41.123747 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:41.123906 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:41.127731 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36293
I20260929 19:02:41.127789 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36293 every 8 connection(s)
I20260929 19:02:41.128114 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:41.129027 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:41.130502 22637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.131980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22569
I20260929 19:02:41.132085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:41.132279 392 external_mini_cluster.cc:1428] Setting key 4ea9780f76b49b93f706ccac36a56652
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260929 19:02:41.140257 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.131064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36293 (local address 127.0.0.1:47222)
0929 19:02:41.131196 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.131228 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.131730 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.132000 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.132003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.132370 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.132596 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.132601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.133681 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.133682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.134367 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.134369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.134878 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.134879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.134942 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.135544 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.135554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.137146 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.138844 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.138850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.138852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.139036 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.139266 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.139267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.139267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.139377 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.139837 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.139841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.139897 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.139966 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.140085 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260929 19:02:41.140290 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.131207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.0.98.62:36293)
0929 19:02:41.131345 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.131349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.131375 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:02:41.131875 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.131877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.131893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.131965 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.132683 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.133587 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.134453 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.134759 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.134890 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.137244 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.137250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.137251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.137257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.138716 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.139125 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.139127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.139128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.139202 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.139473 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.139475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.139476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.139773 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.139847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.140017 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.140158 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260929 19:02:41.140949 22637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.141309 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.141210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:46994)
0929 19:02:41.141250 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.141294 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:02:41.141455 22587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
I20260929 19:02:41.141882 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.141804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47000)
0929 19:02:41.141845 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.141867 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:02:41.142006 22637 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:02:41.171870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40571
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:02:41.191236 22637 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } attempt: 1
I20260929 19:02:41.191725 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.191600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47002)
0929 19:02:41.191656 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.191696 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:02:41.192108 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.192014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47010)
0929 19:02:41.192054 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.192079 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 19:02:41.192193 22637 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
W20260929 19:02:41.252105 22644 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:41.252349 22644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:41.252378 22644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:41.252439 22644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:41.253584 22644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:41.253654 22644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:41.253677 22644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:41.253698 22644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:41.253726 22644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:41.255163 22644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:41.255451 22644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:41.255666 22644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:02:41.260341 22637 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } attempt: 2
W20260929 19:02:41.260535 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:41.260555 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:41.260547 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:41.260798 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.260626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47012)
0929 19:02:41.260727 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.260759 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 19:02:41.260885 22644 server_base.cc:1034] running on GCE node
I20260929 19:02:41.261125 22644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:41.261219 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.261109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47018)
0929 19:02:41.261178 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.261201 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 19:02:41.261333 22637 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40571: connect: Connection refused (error 111)
I20260929 19:02:41.261447 22644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:41.262630 22644 hybrid_clock.cc:648] HybridClock initialized: now 1790708561262612 us; error 30 us; skew 500 ppm
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:41.265127 22644 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:02:41.265537 22644 webserver.cc:460] Webserver started at http://127.0.98.61:46567/ using document root <none> and password file <none>
I20260929 19:02:41.265729 22644 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:41.265776 22644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:41.265863 22644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:41.266852 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data/instance:
uuid: "89e0a8de6e664f3ab1642974b7be658e"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "b732b600250e69ba1ef828edd67402be"
server_key_iv: "40b16ca8bacd56b21e73982273684739"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.267144 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal/instance:
uuid: "89e0a8de6e664f3ab1642974b7be658e"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "b732b600250e69ba1ef828edd67402be"
server_key_iv: "40b16ca8bacd56b21e73982273684739"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.268448 22644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:41.269192 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:41.269361 22644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:41.269435 22644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "89e0a8de6e664f3ab1642974b7be658e"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "b732b600250e69ba1ef828edd67402be"
server_key_iv: "40b16ca8bacd56b21e73982273684739"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.269487 22644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:41.290640 22644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:41.292610 22644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:41.292788 22644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:41.296541 22644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40571
I20260929 19:02:41.296577 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40571 every 8 connection(s)
I20260929 19:02:41.296937 22644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:02:41.297894 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:41.299479 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:41.306814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22644
I20260929 19:02:41.306922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-1/wal/instance
I20260929 19:02:41.307121 392 external_mini_cluster.cc:1428] Setting key 9d189c2a0f24439034d202c7fc5e2894
I20260929 19:02:41.308189 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.300143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.98.62:36293)
0929 19:02:41.300186 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.300188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.300377 (+ 189us) server_negotiation.cc:407] Connection header received
0929 19:02:41.300872 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.300873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.300889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.300940 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.301548 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.302051 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.303028 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.303236 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.303358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.305419 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.305420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.305421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.305428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.306693 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.307191 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.307192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.307192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.307259 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.307548 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.307549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.307550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.307669 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.307738 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.308011 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.308098 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:41.308399 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.300106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36293 (local address 127.0.0.1:47228)
0929 19:02:41.300263 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.300297 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.300762 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.301005 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.301009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.301265 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.301453 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.301457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.302157 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.302158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.302941 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.302943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.303381 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.303382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.303503 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.303990 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.303998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.305335 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.306805 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.306812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.306814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.307118 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.307366 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.307367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.307367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.307481 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.307741 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.307748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.307886 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.307990 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.308147 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260929 19:02:41.308890 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 19:02:41.316133 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.309131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47034)
0929 19:02:41.309347 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.309358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.309407 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.309549 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.309549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.309680 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.309720 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.309721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.310662 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.310663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.311178 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.311180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.311729 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.311729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.311792 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.312215 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.312219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.313287 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.315007 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.315008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.315008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.315168 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.315470 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.315471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.315471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.315531 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.315902 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.315902 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.315939 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.315992 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.316070 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 19:02:41.316394 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.309199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.0.98.61:40571)
0929 19:02:41.309359 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.309364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.309405 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:02:41.309463 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.309465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.309477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.309547 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.309793 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.310576 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.311274 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.311647 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.311730 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.313379 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.313387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.313387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.313393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.314895 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.315271 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.315296 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.315296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.315399 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.315628 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.315629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.315629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.315832 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.315908 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.316015 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.316121 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260929 19:02:41.316932 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 19:02:41.317262 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.317141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60804)
0929 19:02:41.317185 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.317231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:02:41.317399 22662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
I20260929 19:02:41.317889 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.317779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60810)
0929 19:02:41.317829 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.317859 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 19:02:41.318032 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:02:41.335039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35137
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:02:41.352497 22637 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } attempt: 3
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:02:41.359906 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.352914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47044)
0929 19:02:41.352981 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.352993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.353036 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.353188 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.353188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.353362 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.353434 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.353436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.354117 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.354118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.354923 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.354925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.355366 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.355367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.355440 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.355966 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.355971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.357326 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.358877 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.358878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.358879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.359048 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.359261 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.359262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.359262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.359340 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.359582 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.359583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.359628 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.359713 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.359808 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 19:02:41.359906 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.352976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.98.61:40571)
0929 19:02:41.353044 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.353046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.353058 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:02:41.353086 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.353086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.353097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.353169 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.353544 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.354037 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.355006 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.355214 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.355359 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.357429 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.357430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.357431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.357439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.358794 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.359137 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.359138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.359139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.359190 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.359399 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.359400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.359400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.359508 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.359578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.359731 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.359808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:02:41.360347 22637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.360687 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.360594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60826)
0929 19:02:41.360640 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.360669 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:02:41.360985 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.360904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60836)
0929 19:02:41.360943 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.360966 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:02:41.361073 22637 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
I20260929 19:02:41.367255 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } attempt: 1
I20260929 19:02:41.367638 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.367526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60840)
0929 19:02:41.367578 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.367612 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:02:41.368005 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.367909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60848)
0929 19:02:41.367955 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.367983 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:02:41.368120 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
I20260929 19:02:41.392216 22637 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } attempt: 1
I20260929 19:02:41.392657 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.392526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60850)
0929 19:02:41.392590 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.392628 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 19:02:41.393004 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.392914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60858)
0929 19:02:41.392954 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.392979 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:02:41.393087 22637 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
W20260929 19:02:41.417003 22719 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:41.417222 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:41.417240 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:41.417260 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:41.418421 22719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:41.418462 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:41.418475 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:41.418488 22719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:41.418500 22719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:41.420070 22719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36293,127.0.98.61:40571,127.0.98.60:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:41.420320 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:41.420553 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:41.425355 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:41.425374 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:41.425369 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:41.425432 22719 server_base.cc:1034] running on GCE node
I20260929 19:02:41.425963 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:41.426244 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:41.427388 22719 hybrid_clock.cc:648] HybridClock initialized: now 1790708561427369 us; error 40 us; skew 500 ppm
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:41.429793 22719 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:02:41.430171 22719 webserver.cc:460] Webserver started at http://127.0.98.60:45185/ using document root <none> and password file <none>
I20260929 19:02:41.430358 22719 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:41.430408 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:41.430493 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:41.431541 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data/instance:
uuid: "aa5f4df4f79c455b86c55c26f203c0fb"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "fdacb693f99dc04de7d38abb9e1a728f"
server_key_iv: "45a0cf4f76ee74105712c8c0efc9e515"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.431849 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal/instance:
uuid: "aa5f4df4f79c455b86c55c26f203c0fb"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "fdacb693f99dc04de7d38abb9e1a728f"
server_key_iv: "45a0cf4f76ee74105712c8c0efc9e515"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.433050 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:41.433748 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:41.433918 22719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:41.434005 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "aa5f4df4f79c455b86c55c26f203c0fb"
format_stamp: "Formatted at 2026-09-29 19:02:41 on dist-test-slave-gfsd"
server_key: "fdacb693f99dc04de7d38abb9e1a728f"
server_key_iv: "45a0cf4f76ee74105712c8c0efc9e515"
server_key_version: "encryptionkey@0"
I20260929 19:02:41.434077 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:41.436272 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } attempt: 2
I20260929 19:02:41.436638 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.436525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60870)
0929 19:02:41.436578 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.436611 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:02:41.437001 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.436876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60882)
0929 19:02:41.436930 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.436967 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 19:02:41.437124 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35137: connect: Connection refused (error 111)
I20260929 19:02:41.449808 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:41.452203 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:41.452361 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:41.456228 22719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35137
I20260929 19:02:41.456290 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35137 every 8 connection(s)
I20260929 19:02:41.456624 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:02:41.457648 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:41.459148 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.459719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22719
I20260929 19:02:41.459824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708293967505-392-0/minicluster-data/master-2/wal/instance
I20260929 19:02:41.460031 392 external_mini_cluster.cc:1428] Setting key d7869cb9d3b7ea67cdf9a091b43058a5
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:41.468231 22637 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } attempt: 2
I20260929 19:02:41.469401 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.459887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.0.98.62:36293)
0929 19:02:41.459956 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.459958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.460157 (+ 199us) server_negotiation.cc:407] Connection header received
0929 19:02:41.460906 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.460907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.460919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.460999 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.461762 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.462509 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.463754 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.464002 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.464072 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.466385 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.466387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.466387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.466395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.467843 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.468316 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.468317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.468318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.468381 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.468673 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.468674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.468675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.468809 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.468882 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.469219 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.469314 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:02:41.469587 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.459858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36293 (local address 127.0.0.1:47240)
0929 19:02:41.460023 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.460072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.460815 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.461046 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.461049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.461414 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.461664 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.461671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.462588 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.462589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.463680 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.463682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.464099 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.464100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.464201 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.464668 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.464679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.466304 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.467919 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.467926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.467928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.468211 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.468457 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.468458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.468458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.468587 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.468884 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.468889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.469054 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.469167 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.469339 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260929 19:02:41.470134 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.470441 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.460425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.0.98.60:35137)
0929 19:02:41.460591 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.460595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.460618 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:02:41.460832 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.460842 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.460880 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.460994 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.461768 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.462859 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.463891 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.464272 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.464323 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.466962 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.466969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.466969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.466975 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.468966 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.469350 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.469353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.469353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.469410 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.469595 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.469595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.469596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.469737 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.469864 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.470257 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.470333 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:41.478029 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.470449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40571 (local address 127.0.0.1:47046)
0929 19:02:41.470488 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.470496 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.470531 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.470671 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.470672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.470803 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.470848 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.470849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.471654 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.471655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.472491 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.472494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.472964 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.472965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.473046 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.473453 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.473459 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.474816 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.476859 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.476860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.476861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.477109 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.477314 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.477315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.477315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.477395 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.477678 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.477679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.477731 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.477833 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.477920 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 19:02:41.478181 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.468474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60896)
0929 19:02:41.468524 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.468535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.468579 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.468949 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.468950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.469101 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.469141 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.469142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.470043 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.470044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.470860 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.470862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.471217 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.471218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.471306 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.472577 (+ 1271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.472582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.474113 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.476586 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.476587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.476587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.476758 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.477183 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.477184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.477184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.477252 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.477828 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.477829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.477874 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.477922 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.478030 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:02:41.478446 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.470508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.0.98.61:40571)
0929 19:02:41.470553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.470555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.470564 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:41.470589 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.470589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.470611 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.470667 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.470944 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.471565 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.472559 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.472886 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.472955 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.474903 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.474905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.474906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.474913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.476777 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.477184 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.477185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.477185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.477243 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.477465 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.477466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.477466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.477598 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.477676 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.478211 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.478301 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:02:41.478575 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.468654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.0.98.60:35137)
0929 19:02:41.468845 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.468847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.468855 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:41.468873 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.468874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.468883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.468946 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.469228 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.469955 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.470945 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.471145 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.471191 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.474206 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.474208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.474208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.474216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.476508 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.476839 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.476840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.476840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.476906 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.477327 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.477328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.477328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.477742 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.478286 (+ 544us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.478332 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.478424 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":116,"threads_started":1}
I20260929 19:02:41.478791 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:41.479893 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.471679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.98.62:36293)
0929 19:02:41.471716 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.471718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.471749 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:02:41.471816 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.471816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.471825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.471869 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.472423 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.473174 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.473898 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.474084 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.474176 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.476755 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.476756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.476757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.476764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.478424 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.478809 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.478810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.478811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.478865 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.479070 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.479071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.479071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.479206 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.479368 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.479706 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.479782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:41.480464 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: Bootstrap starting.
I20260929 19:02:41.481199 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:41.481475 22637 log.cc:824] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:41.482277 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: No bootstrap required, opened a new log
I20260929 19:02:41.484293 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.484436 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4365e8496a4382894cabef1dc77ec3, State: Initialized, Role: FOLLOWER
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:41.484548 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.484916 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:02:41.485538 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.485723 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:02:41.486744 22805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:41.487152 22805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:41.488143 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.479012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60902)
0929 19:02:41.479107 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.479116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.479156 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.479262 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.479262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.479415 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.479440 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.479441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.480442 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.480443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.481155 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.481158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.481641 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.481642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.481700 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.483393 (+ 1693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.483398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.484844 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.487063 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.487064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.487064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.487263 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.487465 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.487466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.487466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.487516 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.487835 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.487835 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.487874 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.487927 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.488013 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 19:02:41.488704 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.479076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.0.98.60:35137)
0929 19:02:41.479110 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.479112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.479143 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:02:41.479205 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.479206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.479214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.479261 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.479511 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.480374 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.481233 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.481559 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.481627 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.484944 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.484945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.484945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.484951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.486977 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.487356 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.487357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.487357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.487406 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.487573 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.487573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.487574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.487754 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.487814 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.488529 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.488605 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:41.489069 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.480561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.0.98.61:40571)
0929 19:02:41.480594 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.480596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.480602 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:02:41.480619 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.480619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.480627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.480673 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.481112 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.481802 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.482766 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.483029 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.483107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.485847 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.485849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.485849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.485856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.487738 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.488123 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.488124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.488124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.488192 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.488421 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.488421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.488422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.488542 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.488669 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.488900 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.488973 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:02:41.490557 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: Bootstrap starting.
I20260929 19:02:41.490723 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:41.487985 22569 master_runner.cc:186] Error getting master registration for 127.0.98.61:40571: OK, instance_id { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" instance_seqno: 1790708561295438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:02:41.490903 22569 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:02:41.491428 22787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:41.491787 22787 log.cc:824] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:41.491901 22611 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4a4365e8496a4382894cabef1dc77ec3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242
I20260929 19:02:41.492030 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:02:41.492089 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:41.492610 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: No bootstrap required, opened a new log
I20260929 19:02:41.492769 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.492966 22611 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [CANDIDATE]: Term 1 election: Requested vote from peers 89e0a8de6e664f3ab1642974b7be658e (127.0.98.61:40571), aa5f4df4f79c455b86c55c26f203c0fb (127.0.98.60:35137)
I20260929 19:02:41.494604 22787 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.494745 22787 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5f4df4f79c455b86c55c26f203c0fb, State: Initialized, Role: FOLLOWER
I20260929 19:02:41.494829 22787 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.495117 22787 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:41.495182 22761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a4365e8496a4382894cabef1dc77ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa5f4df4f79c455b86c55c26f203c0fb"
I20260929 19:02:41.495352 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:41.496703 22761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4365e8496a4382894cabef1dc77ec3 in term 1.
I20260929 19:02:41.497860 22587 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4365e8496a4382894cabef1dc77ec3, aa5f4df4f79c455b86c55c26f203c0fb; no voters:
I20260929 19:02:41.497996 22794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:41.498119 22794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4365e8496a4382894cabef1dc77ec3, State: Running, Role: LEADER
I20260929 19:02:41.498194 22794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.498607 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4365e8496a4382894cabef1dc77ec3. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.498704 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:41.501144 22815 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.502471 22815 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:41.500540 22719 master_runner.cc:186] Error getting master registration for 127.0.98.61:40571: OK, instance_id { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" instance_seqno: 1790708561295438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:02:41.502852 22719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:02:41.505177 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.494069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.0.98.61:40571)
0929 19:02:41.494108 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.494109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.494612 (+ 503us) server_negotiation.cc:407] Connection header received
0929 19:02:41.494706 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.494707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.494718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.494770 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.496830 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.497511 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.498547 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.498839 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.498882 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.500589 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.500591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.500591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.500598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.502272 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.502664 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.502665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.502665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.502715 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.502903 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.502903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.502904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.503021 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.503817 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.504039 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.505083 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:41.505209 22829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:41.505719 22829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:41.506536 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.494239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.98.60:35137)
0929 19:02:41.494273 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.494275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.494758 (+ 483us) server_negotiation.cc:407] Connection header received
0929 19:02:41.495726 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.495727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.495735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.495764 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.496649 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.497395 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.498442 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.498722 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.498772 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.503320 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.503322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.503322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.503328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.504957 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.505729 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.505730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.505730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.505783 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.505969 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.505970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.505970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.506083 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.506159 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.506361 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.506453 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:02:41.508057 22761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [term 1 FOLLOWER]: Refusing update from remote peer 4a4365e8496a4382894cabef1dc77ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 19:02:41.508039 22826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:41.508301 22826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:41.508421 22827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:41.509668 22815 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4365e8496a4382894cabef1dc77ec3. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.509765 22815 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:41.510298 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.494483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.0.98.62:36293)
0929 19:02:41.494522 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.494524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.497590 (+ 3066us) server_negotiation.cc:407] Connection header received
0929 19:02:41.497613 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.497614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.497624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.497681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.501400 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.502133 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.503973 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.504271 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.504313 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.506794 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.506796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.506796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.506804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.508687 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.509074 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.509075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.509075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.509147 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.509407 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.509408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.509409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.509550 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.509650 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.510126 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.510214 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:41.510964 22830 mvcc.cc:205] Tried to move back new op lower bound from 7334742267933986816 to 7334742267897430016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742267933986816}]
I20260929 19:02:41.511085 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.511161 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:41.511508 22815 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.511605 22815 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:41.515630 22829 catalog_manager.cc:1347] Generated new cluster ID: 1ae98a2b3d3b468ea2addfe29d95e1d9
I20260929 19:02:41.515686 22829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:41.518359 22832 mvcc.cc:205] Tried to move back new op lower bound from 7334742267933986816 to 7334742267897430016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742267933986816}]
I20260929 19:02:41.528260 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } attempt: 3
I20260929 19:02:41.528963 22829 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:41.529662 22829 catalog_manager.cc:1504] Loading token signing keys...
Sep 29 19:02:41 dist-test-slave-gfsd krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:41.535997 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.528549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.0.98.60:35137)
0929 19:02:41.528587 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.528589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.528598 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:41.528659 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.528674 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.528684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.528744 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.529135 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.529868 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.530887 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.531196 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.531266 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.533328 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:41.533329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:41.533330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:41.533336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:41.534857 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.535230 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.535231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.535231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.535312 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:41.535511 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:41.535512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:41.535512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:41.535632 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:41.535689 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:41.535839 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:02:41.535908 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:02:41.536000 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:41.528502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35137 (local address 127.0.0.1:60932)
0929 19:02:41.528561 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:41.528572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:41.528608 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:41.528794 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:41.528796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:41.528977 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:41.529023 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.529025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.529972 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.529973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.530814 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:41.530816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:41.531263 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:41.531264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:41.531351 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:41.531796 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:41.531801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:41.533228 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:41.534939 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.534940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.534940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.535156 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.535373 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:41.535374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:41.535374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:41.535435 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:41.535677 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:41.535678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:41.535732 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 19:02:41.535804 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:02:41.535898 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:02:41.537892 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: Bootstrap starting.
I20260929 19:02:41.538610 22712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:41.538904 22712 log.cc:824] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:41.540472 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: No bootstrap required, opened a new log
I20260929 19:02:41.542490 22712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.542623 22712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0a8de6e664f3ab1642974b7be658e, State: Initialized, Role: FOLLOWER
I20260929 19:02:41.542702 22712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } }
I20260929 19:02:41.543068 22686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a4365e8496a4382894cabef1dc77ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "89e0a8de6e664f3ab1642974b7be658e"
I20260929 19:02:41.543109 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:41.543020 22712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:41.544405 22685 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [term 1 FOLLOWER]: Refusing update from remote peer 4a4365e8496a4382894cabef1dc77ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:41.544781 22827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:41.545892 22686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4365e8496a4382894cabef1dc77ec3 in term 1.
I20260929 19:02:41.543013 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
W20260929 19:02:41.543330 22665 tablet.cc:2369] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:41.547873 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:41.548007 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4365e8496a4382894cabef1dc77ec3. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.548107 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:41.548187 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.548267 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:02:41.548849 22848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:41.548905 22848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:41.550060 22850 mvcc.cc:205] Tried to move back new op lower bound from 7334742268016148480 to 7334742267897430016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742267933986816}]
I20260929 19:02:41.550678 22829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3: Generated new TSK 0
I20260929 19:02:41.550892 22829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:41.551151 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a4365e8496a4382894cabef1dc77ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4365e8496a4382894cabef1dc77ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36293 } } peers { permanent_uuid: "89e0a8de6e664f3ab1642974b7be658e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40571 } } peers { permanent_uuid: "aa5f4df4f79c455b86c55c26f203c0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35137 } } }
I20260929 19:02:41.551249 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4365e8496a4382894cabef1dc77ec3 [sys.catalog]: This master's current role is: LEADER
W20260929 19:02:41.581122 22640 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.579140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.0.98.62:36293)
0929 19:02:41.579187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.579189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.580297 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 19:02:41.580403 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.580404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.580414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.580446 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.581050 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47254: BlockingRecv error: recv got EOF from 127.0.0.1:47254 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 19:02:41.581929 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.579389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.0.98.61:40571)
0929 19:02:41.579429 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.579431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.581385 (+ 1954us) server_negotiation.cc:407] Connection header received
0929 19:02:41.581498 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.581499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.581510 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.581539 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.581854 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47084: BlockingRecv error: recv got EOF from 127.0.0.1:47084 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:02:41.582749 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.580021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.0.98.60:35137)
0929 19:02:41.580060 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.580062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.582154 (+ 2092us) server_negotiation.cc:407] Connection header received
0929 19:02:41.582264 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.582266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.582277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.582306 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.582601 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60936: BlockingRecv error: recv got EOF from 127.0.0.1:60936 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:02:41.585109 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.584340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.0.98.61:40571)
0929 19:02:41.584375 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.584376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.584587 (+ 211us) server_negotiation.cc:407] Connection header received
0929 19:02:41.584703 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.584704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.584721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.584749 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.585053 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47096: BlockingRecv error: recv got EOF from 127.0.0.1:47096 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:02:41.588186 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.587397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.98.61:40571)
0929 19:02:41.587429 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.587431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.587702 (+ 271us) server_negotiation.cc:407] Connection header received
0929 19:02:41.587797 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.587798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.587807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.587835 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.588138 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47104: BlockingRecv error: recv got EOF from 127.0.0.1:47104 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:02:41.592199 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.591442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.0.98.61:40571)
0929 19:02:41.591477 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.591478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.591719 (+ 241us) server_negotiation.cc:407] Connection header received
0929 19:02:41.591810 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.591811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.591820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.591848 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.592156 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47110: BlockingRecv error: recv got EOF from 127.0.0.1:47110 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:02:41.598052 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.597305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.0.98.61:40571)
0929 19:02:41.597338 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.597339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.597573 (+ 234us) server_negotiation.cc:407] Connection header received
0929 19:02:41.597666 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.597666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.597676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.597704 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.598004 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47112: BlockingRecv error: recv got EOF from 127.0.0.1:47112 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:02:41.608131 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.607206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.98.61:40571)
0929 19:02:41.607245 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.607247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.607586 (+ 339us) server_negotiation.cc:407] Connection header received
0929 19:02:41.607682 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.607683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.607694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.607723 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.608070 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47128: BlockingRecv error: recv got EOF from 127.0.0.1:47128 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:02:41.626430 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.625460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.98.61:40571)
0929 19:02:41.625501 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.625504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.625795 (+ 291us) server_negotiation.cc:407] Connection header received
0929 19:02:41.625909 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.625910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.625923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.625956 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.626369 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47140: BlockingRecv error: recv got EOF from 127.0.0.1:47140 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:02:41.660830 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.659829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.98.61:40571)
0929 19:02:41.659876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.659879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.660199 (+ 320us) server_negotiation.cc:407] Connection header received
0929 19:02:41.660317 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.660318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.660331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.660363 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.660771 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47142: BlockingRecv error: recv got EOF from 127.0.0.1:47142 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 19:02:41.727013 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.726198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.0.98.61:40571)
0929 19:02:41.726265 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.726267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.726402 (+ 135us) server_negotiation.cc:407] Connection header received
0929 19:02:41.726487 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.726488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.726506 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.726559 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.726931 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47150: BlockingRecv error: recv got EOF from 127.0.0.1:47150 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 19:02:41.856892 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:41.856201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.0.98.61:40571)
0929 19:02:41.856269 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:41.856273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:41.856288 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:02:41.856334 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:41.856334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:41.856344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:41.856407 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:41.856795 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47166: BlockingRecv error: recv got EOF from 127.0.0.1:47166 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 19:02:42.114909 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:42.114117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.0.98.61:40571)
0929 19:02:42.114184 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.114186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.114201 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:02:42.114303 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.114304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.114342 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.114410 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:42.114834 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47168: BlockingRecv error: recv got EOF from 127.0.0.1:47168 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:02:42.508744 22826 catalog_manager.cc:1259] Loaded cluster ID: 1ae98a2b3d3b468ea2addfe29d95e1d9
I20260929 19:02:42.508843 22826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: loading cluster ID for follower catalog manager: success
I20260929 19:02:42.509662 22826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: acquiring CA information for follower catalog manager: success
I20260929 19:02:42.509985 22826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa5f4df4f79c455b86c55c26f203c0fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:02:42.549330 22848 catalog_manager.cc:1259] Loaded cluster ID: 1ae98a2b3d3b468ea2addfe29d95e1d9
I20260929 19:02:42.549417 22848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: loading cluster ID for follower catalog manager: success
I20260929 19:02:42.550366 22848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: acquiring CA information for follower catalog manager: success
I20260929 19:02:42.550827 22848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89e0a8de6e664f3ab1642974b7be658e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:02:42.630900 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:42.628157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.0.98.61:40571)
0929 19:02:42.628284 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.628286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.628296 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:42.628343 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.628344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.628390 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.628455 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:42.628721 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.629239 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.630077 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.630292 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.630364 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:42.630555 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:42.630665 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:42.630729 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:42.630818 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:02:42.630835 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260929 19:02:42.635583 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:42.633002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.0.98.60:35137)
0929 19:02:42.633142 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.633145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.633154 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:42.633201 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.633202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.633241 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.633331 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:42.633432 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.633963 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.634708 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.634937 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.635013 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:42.635188 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:42.635348 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:42.635423 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:42.635460 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 19:02:42.635482 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":48,"threads_started":1}
I20260929 19:02:42.640327 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:42.637649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.0.98.60:35137)
0929 19:02:42.637683 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.637684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.637698 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:02:42.637717 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.637717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.637727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.637815 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:42.637962 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.638678 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.639526 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.639772 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.639849 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:42.640004 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:42.640037 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:42.640083 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:42.640182 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:02:42.640210 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:42.640434 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:42.637259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.0.98.61:40571)
0929 19:02:42.637301 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.637303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.637540 (+ 237us) server_negotiation.cc:407] Connection header received
0929 19:02:42.637642 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.637643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.637654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.637695 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:42.637884 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.638403 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.639476 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.639769 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.639855 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:42.640175 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:42.640207 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:42.640259 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:42.640317 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:02:42.640343 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:02:42.640679 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:42.637290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.0.98.62:36293)
0929 19:02:42.637394 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:42.637397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:42.637406 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:42.637476 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:42.637477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:42.637531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:42.637753 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:42.637794 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.638577 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.639617 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:42.639931 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:42.640008 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:42.640188 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:42.640303 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:42.640530 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:42.640550 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 19:02:42.640578 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260929 19:02:42.642637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22569
I20260929 19:02:42.648952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22644
I20260929 19:02:42.654420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22719
2026-09-29T19:02:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2802 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:42 dist-test-slave-gfsd krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:42 dist-test-slave-gfsd krb5kdc[22938](info): set up 2 sockets
Sep 29 19:02:42 dist-test-slave-gfsd krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260929 19:02:43.719596 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:43 dist-test-slave-gfsd krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:43.840183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:43.920475 22951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:43.920722 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:43.920753 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:43.920797 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:43.921936 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:43.922003 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:43.922026 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:43.922047 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:43.922077 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:43.923556 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:43.923828 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:43.924050 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:02:43.929152 22951 server_base.cc:1034] running on GCE node
W20260929 19:02:43.929107 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:43.929170 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:43.929105 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:43.929551 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:43.929812 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:43.930950 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790708563930942 us; error 33 us; skew 500 ppm
Sep 29 19:02:43 dist-test-slave-gfsd krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:43.933429 22951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:43.933879 22951 webserver.cc:460] Webserver started at http://127.0.98.62:44061/ using document root <none> and password file <none>
I20260929 19:02:43.934105 22951 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:43.934173 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:43.934327 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:43.935449 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data/instance:
uuid: "b1f0ab0c996e441fbb26366244e2e101"
format_stamp: "Formatted at 2026-09-29 19:02:43 on dist-test-slave-gfsd"
server_key: "88faa5430a01925531aded78f5ed6963"
server_key_iv: "fb899ae272ff1abfc0501d0089f3f697"
server_key_version: "encryptionkey@0"
I20260929 19:02:43.935807 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1f0ab0c996e441fbb26366244e2e101"
format_stamp: "Formatted at 2026-09-29 19:02:43 on dist-test-slave-gfsd"
server_key: "88faa5430a01925531aded78f5ed6963"
server_key_iv: "fb899ae272ff1abfc0501d0089f3f697"
server_key_version: "encryptionkey@0"
I20260929 19:02:43.937104 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:43.937837 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:43.938019 22951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:43.938123 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal
uuid: "b1f0ab0c996e441fbb26366244e2e101"
format_stamp: "Formatted at 2026-09-29 19:02:43 on dist-test-slave-gfsd"
server_key: "88faa5430a01925531aded78f5ed6963"
server_key_iv: "fb899ae272ff1abfc0501d0089f3f697"
server_key_version: "encryptionkey@0"
I20260929 19:02:43.938197 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:43.950681 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:43.952637 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:43.952800 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:43.956858 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36837
I20260929 19:02:43.956893 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36837 every 8 connection(s)
I20260929 19:02:43.957239 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:43.958164 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:43.959659 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:02:43 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:43.965301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 22951
I20260929 19:02:43.965406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:43.965579 392 external_mini_cluster.cc:1428] Setting key a2d08f69202bb87f1b87c752dfc74349
I20260929 19:02:43.969084 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:43.960238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.0.1:34816)
0929 19:02:43.960403 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:43.960437 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:43.960862 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:43.961104 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:43.961106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:43.961353 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:43.961504 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:43.961508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:43.962467 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:43.962467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:43.963117 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:43.963119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:43.963640 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:43.963641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:43.963698 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:43.964198 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:43.964206 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:43.965801 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:43.967638 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:43.967644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:43.967646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:43.967838 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:43.968078 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:43.968079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:43.968079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:43.968165 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:43.968512 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:43.968517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:43.968576 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 19:02:43.968680 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 19:02:43.968907 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":93,"thread_start_us":67,"threads_started":1}
I20260929 19:02:43.969087 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:43.960387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.98.62:36837)
0929 19:02:43.960505 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:43.960509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:43.960548 (+ 39us) server_negotiation.cc:407] Connection header received
0929 19:02:43.960984 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:43.960985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:43.961001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:43.961078 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:43.961580 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:43.962385 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:43.963186 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:43.963555 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:43.963633 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:43.965904 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:43.965911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:43.965911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:43.965924 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:43.967545 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:43.967931 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:43.967934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:43.967935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:43.968010 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:43.968229 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:43.968230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:43.968230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:43.968437 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:43.968522 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:43.968705 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 19:02:43.968861 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260929 19:02:43.969769 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:43.970126 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:43.970000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38234)
0929 19:02:43.970045 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:43.970096 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:02:43.971073 22967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
I20260929 19:02:43.971551 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:43.971483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38246)
0929 19:02:43.971510 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:43.971534 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 19:02:43.971668 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40275: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:02:43.990742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40275
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:02:44.020893 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } attempt: 1
I20260929 19:02:44.021459 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.021278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38256)
0929 19:02:44.021368 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.021422 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 19:02:44.022008 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.021899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38268)
0929 19:02:44.021952 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.021983 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 19:02:44.022126 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40275: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
W20260929 19:02:44.070255 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:44.070520 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:44.070550 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:44.070590 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:44.071729 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:44.071798 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:44.071825 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:44.071844 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:44.071861 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:44.073328 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:44.073603 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:44.073815 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:44.078794 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:44.078794 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:44.078785 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:44.078848 23026 server_base.cc:1034] running on GCE node
I20260929 19:02:44.079259 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:44.079560 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:44.080756 23026 hybrid_clock.cc:648] HybridClock initialized: now 1790708564080745 us; error 36 us; skew 500 ppm
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:44.083258 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:02:44.083748 23026 webserver.cc:460] Webserver started at http://127.0.98.61:42921/ using document root <none> and password file <none>
I20260929 19:02:44.084003 23026 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:44.084070 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:44.084221 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:44.085294 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data/instance:
uuid: "77c1fc0350314f28b019d388f379b68f"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "1513b881543997c8c9ce217e2d976007"
server_key_iv: "a529e105e7a9286a0429e636cb71aed6"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.085646 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal/instance:
uuid: "77c1fc0350314f28b019d388f379b68f"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "1513b881543997c8c9ce217e2d976007"
server_key_iv: "a529e105e7a9286a0429e636cb71aed6"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.086961 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:44.087741 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:44.087917 23026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:44.088011 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal
uuid: "77c1fc0350314f28b019d388f379b68f"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "1513b881543997c8c9ce217e2d976007"
server_key_iv: "a529e105e7a9286a0429e636cb71aed6"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.088088 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:44.090266 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } attempt: 2
I20260929 19:02:44.090632 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.090514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38276)
0929 19:02:44.090572 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.090606 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:02:44.090943 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.090854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38282)
0929 19:02:44.090898 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.090922 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:02:44.091032 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40275: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40275: connect: Connection refused (error 111)
I20260929 19:02:44.106881 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:44.109261 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:44.109421 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:44.113351 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40275
I20260929 19:02:44.113415 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40275 every 8 connection(s)
I20260929 19:02:44.113773 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:02:44.114705 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:44.115343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 23026
I20260929 19:02:44.115447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-1/wal/instance
I20260929 19:02:44.115638 392 external_mini_cluster.cc:1428] Setting key 3f3992ab7e13bde2e3e40b5407bd4a2d
I20260929 19:02:44.117015 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 19:02:44.127441 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.117839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.0.98.62:36837)
0929 19:02:44.117896 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.117898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.118100 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:02:44.118601 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.118603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.118615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.118673 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.119261 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.119767 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.120745 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.120955 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.121028 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.123340 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.123342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.123343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.123351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.125006 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.125377 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.125378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.125378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.125455 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.125729 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.125731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.125731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.125856 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.125935 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.127229 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.127346 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:02:44.127667 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.117813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.0.1:34822)
0929 19:02:44.117985 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.118018 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.118495 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.118744 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.118748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.118983 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.119164 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.119168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.119877 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.119878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.120639 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.120640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.121047 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.121048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.121170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.121705 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.121713 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.123165 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.125096 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.125102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.125104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.125313 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.125533 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.125534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.125534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.125666 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.126957 (+ 1291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.126963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.127085 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.127207 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.127394 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260929 19:02:44.128222 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:02:44.136042 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.128477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38292)
0929 19:02:44.128523 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.128534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.128569 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.128842 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.128843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.129011 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.129078 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.129079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.129938 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.129939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.130864 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.130866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.131348 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.131349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.131432 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.131811 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.131817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.133078 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.134868 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.134877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.134878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.135039 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.135233 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.135234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.135234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.135330 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.135752 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.135760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.135800 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.135875 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.135975 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:02:44.136206 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.128570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.61:40275)
0929 19:02:44.128711 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.128716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.128730 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:02:44.128744 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.128746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.128756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.128807 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.129173 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.129852 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.130945 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.131233 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.131333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.133165 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.133173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.133174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.133182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.134767 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.135113 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.135115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.135115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.135169 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.135398 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.135399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.135399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.135585 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.135751 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.135913 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.136025 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260929 19:02:44.136857 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:44.137218 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.137084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52062)
0929 19:02:44.137155 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.137200 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 19:02:44.137480 23043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
I20260929 19:02:44.138031 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.137913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52074)
0929 19:02:44.137981 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.138012 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 19:02:44.138213 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:02:44.142097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHoo0t8/build/release/bin/kudu
/tmp/dist-test-taskHoo0t8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39077
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:02:44.182190 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } attempt: 3
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:02:44.187422 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } attempt: 1
I20260929 19:02:44.187805 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.187670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52088)
0929 19:02:44.187729 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.187767 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:02:44.188199 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.188101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52096)
0929 19:02:44.188147 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.188174 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:02:44.188396 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
I20260929 19:02:44.189580 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.182636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38308)
0929 19:02:44.182721 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.182735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.182824 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.182979 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.182980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.183149 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.183216 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.183218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.184028 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.184029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.184614 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.184617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.184983 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.184984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.185056 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.185535 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.185541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.186742 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.188625 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.188626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.188626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.188794 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.188988 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.188989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.188989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.189038 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.189326 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.189327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.189372 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.189441 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.189517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 19:02:44.189640 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.182776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.0.98.61:40275)
0929 19:02:44.182845 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.182847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.182858 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:02:44.182892 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.182896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.182908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.182974 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.183326 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.183886 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.184704 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.184909 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.184986 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.186831 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.186832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.186833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.186840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.188535 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.188861 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.188862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.188863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.188925 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.189094 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.189095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.189095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.189221 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.189286 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.189481 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.189565 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:02:44.190354 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:44.190754 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.190595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52102)
0929 19:02:44.190707 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.190733 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:02:44.191073 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.190986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52112)
0929 19:02:44.191031 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.191054 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:02:44.191174 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
I20260929 19:02:44.222333 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } attempt: 1
I20260929 19:02:44.222790 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.222634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52120)
0929 19:02:44.222710 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.222754 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 19:02:44.223187 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.223093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52130)
0929 19:02:44.223137 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.223166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:02:44.223301 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
W20260929 19:02:44.223321 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:02:44.223523 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:44.223539 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:44.223559 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:44.224658 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:44.224714 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:44.224728 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:44.224741 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:44.224753 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:44.226205 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36837,127.0.98.61:40275,127.0.98.60:39077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:44.226385 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:44.226547 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:44.231300 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:44.231356 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:44.231303 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:44.231354 23101 server_base.cc:1034] running on GCE node
I20260929 19:02:44.231834 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:44.232120 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:44.233266 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790708564233259 us; error 46 us; skew 500 ppm
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:44.235889 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:02:44.236373 23101 webserver.cc:460] Webserver started at http://127.0.98.60:34383/ using document root <none> and password file <none>
I20260929 19:02:44.236608 23101 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:44.236656 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:44.236781 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:44.237852 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data/instance:
uuid: "530bccf2fc684bf6bd55c548dbf0aa3d"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "5d3703fdec997924864f9e95222f4d8b"
server_key_iv: "e2533dc6fede738af4e89f22a1d150e1"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.238210 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal/instance:
uuid: "530bccf2fc684bf6bd55c548dbf0aa3d"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "5d3703fdec997924864f9e95222f4d8b"
server_key_iv: "e2533dc6fede738af4e89f22a1d150e1"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.239593 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:44.240496 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:44.240727 23101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:44.240828 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal
uuid: "530bccf2fc684bf6bd55c548dbf0aa3d"
format_stamp: "Formatted at 2026-09-29 19:02:44 on dist-test-slave-gfsd"
server_key: "5d3703fdec997924864f9e95222f4d8b"
server_key_iv: "e2533dc6fede738af4e89f22a1d150e1"
server_key_version: "encryptionkey@0"
I20260929 19:02:44.240907 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:44.254559 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:44.256546 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } attempt: 2
I20260929 19:02:44.256631 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:44.256781 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:44.256970 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.256841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52138)
0929 19:02:44.256901 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.256943 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:02:44.257411 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.257294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52146)
0929 19:02:44.257350 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.257383 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 19:02:44.257548 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39077: connect: Connection refused (error 111)
I20260929 19:02:44.261209 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39077
I20260929 19:02:44.261240 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39077 every 8 connection(s)
I20260929 19:02:44.261582 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:02:44.262567 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:44.264163 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:44.266727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHoo0t8/build/release/bin/kudu as pid 23101
I20260929 19:02:44.266839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHoo0t8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708293967505-392-0/minicluster-data/master-2/wal/instance
I20260929 19:02:44.267040 392 external_mini_cluster.cc:1428] Setting key 771d29d7c6b3530eac65b4bf080567a1
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:44.274569 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.264777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.62:36837)
0929 19:02:44.264836 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.264838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.265023 (+ 185us) server_negotiation.cc:407] Connection header received
0929 19:02:44.265482 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.265483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.265492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.265543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.266190 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.266790 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.268119 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.268437 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.268502 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.271028 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.271029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.271029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.271036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.273150 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.273619 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.273620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.273620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.273681 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.273918 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.273919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.273919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.274049 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.274122 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.274403 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.274477 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:02:44.274801 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.264746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.0.1:34832)
0929 19:02:44.264904 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.264944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.265384 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.265613 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.265616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.265918 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.266104 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.266109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.266919 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.266920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.268041 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.268044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.268553 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.268554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.268704 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.269210 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.269220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.270944 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.273237 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.273243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.273245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.273552 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.273752 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.273753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.273753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.273851 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.274129 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.274136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.274271 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.274381 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.274534 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260929 19:02:44.275400 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:44.277285 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.267426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.98.60:39077)
0929 19:02:44.267584 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.267588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.267648 (+ 60us) server_negotiation.cc:407] Connection header received
0929 19:02:44.267721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.267723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.267745 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.267818 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.268518 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.269626 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.270715 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.271141 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.271191 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.273974 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.273977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.273977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.273981 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.275973 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.276332 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.276336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.276336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.276411 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.276597 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.276598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.276598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.276823 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.276998 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.277105 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.277217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:44.283104 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.275752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.61:40275)
0929 19:02:44.275802 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.275803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.275812 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:44.275857 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.275857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.275867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.275921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.276223 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.277037 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.278109 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.278429 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.278484 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.280337 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.280339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.280339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.280346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.281916 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.282328 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.282329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.282330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.282378 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.282586 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.282588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.282588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.282711 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.282793 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.282946 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.283017 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:02:44.283118 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.275675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40275 (local address 127.0.0.1:38310)
0929 19:02:44.275754 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.275762 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.275798 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.275925 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.275925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.276080 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.276112 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.276114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.277229 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.277230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.278029 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.278031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.278503 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.278504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.278575 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.279053 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.279058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.280234 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.282004 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.282005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.282005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.282192 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.282444 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.282445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.282446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.282515 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.282785 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.282786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.282829 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.282920 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.283012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 19:02:44.283630 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:02:44.285295 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.278174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.0.98.62:36837)
0929 19:02:44.278212 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.278214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.278221 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:02:44.278241 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.278242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.278251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.278318 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.278843 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.279318 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.279942 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.280209 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.280286 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.282265 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.282266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.282266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.282274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.284041 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.284533 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.284534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.284534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.284582 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.284860 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.284861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.284861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.284943 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.285053 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.285130 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.285189 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:02:44.291191 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.283986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.98.60:39077)
0929 19:02:44.284038 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.284040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.284058 (+ 18us) server_negotiation.cc:407] Connection header received
0929 19:02:44.284097 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.284097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.284107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.284155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.284444 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.285173 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.285987 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.286321 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.286394 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.288431 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.288432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.288432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.288438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.289997 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.290336 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.290337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.290338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.290415 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.290647 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.290648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.290648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.290749 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.290823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.291004 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.291083 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:44.291452 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.283935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52164)
0929 19:02:44.283976 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.283984 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.284019 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.284172 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.284173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.284318 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.284365 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.284367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.285250 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.285252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.285926 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.285928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.286410 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.286411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.286454 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.286834 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.286838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.288356 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.290078 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.290079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.290079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.290266 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.290491 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.290492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.290492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.290546 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.290816 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.290816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.290852 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.290994 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.291340 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":241}
I20260929 19:02:44.293375 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: Bootstrap starting.
I20260929 19:02:44.293797 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.285895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.0.98.61:40275)
0929 19:02:44.285929 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.285931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.285990 (+ 59us) server_negotiation.cc:407] Connection header received
0929 19:02:44.286076 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.286077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.286085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.286120 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.286701 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.287464 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.288331 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.288586 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.288668 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.290877 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.290878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.290878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.290885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.292569 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.292975 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.292976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.292977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.293039 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.293245 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.293246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.293246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.293375 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.293533 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.293619 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.293690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:02:44.294457 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:44.294821 23169 log.cc:824] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:44.295342 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:02:44.295692 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: No bootstrap required, opened a new log
I20260929 19:02:44.296278 22993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b1f0ab0c996e441fbb26366244e2e101"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848
I20260929 19:02:44.297362 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.297505 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bccf2fc684bf6bd55c548dbf0aa3d, State: Initialized, Role: FOLLOWER
I20260929 19:02:44.297595 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.297938 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:44.297914 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.297991 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.298436 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } attempt: 2
W20260929 19:02:44.299736 23186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:44.299788 23186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:44.300370 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:36837: OK, instance_id { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" instance_seqno: 1790708563955750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:02:44.303200 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:02:44.305814 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.298688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52174)
0929 19:02:44.298735 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.298747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.298787 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.298944 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.298945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.299117 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.299166 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.299169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.300021 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.300022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.300813 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.300816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.301177 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.301178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.301237 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.301663 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.301685 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.303084 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.304761 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.304762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.304762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.304942 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.305159 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.305160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.305161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.305231 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.305500 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.305501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.305544 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.305624 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.305724 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 19:02:44.305814 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.298745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.98.60:39077)
0929 19:02:44.298783 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.298785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.298794 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:44.298841 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.298842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.298851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.298909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.299256 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.299934 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.300885 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.301097 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.301173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.303163 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.303164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.303164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.303171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.304643 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.305026 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.305027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.305028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.305089 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.305295 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.305296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.305296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.305424 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.305495 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.305662 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.305735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:44.307302 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: Bootstrap starting.
I20260929 19:02:44.307976 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:44.308266 23019 log.cc:824] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:44.308982 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: No bootstrap required, opened a new log
I20260929 19:02:44.309949 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.310051 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f0ab0c996e441fbb26366244e2e101, State: Initialized, Role: FOLLOWER
I20260929 19:02:44.310132 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.310398 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.310469 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.310472 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:44.310525 22993 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:02:44.310597 22993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:44.311486 22993 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.311721 22993 leader_election.cc:288] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [CANDIDATE]: Term 1 election: Requested vote from peers 77c1fc0350314f28b019d388f379b68f (127.0.98.61:40275), 530bccf2fc684bf6bd55c548dbf0aa3d (127.0.98.60:39077)
I20260929 19:02:44.312294 23143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1f0ab0c996e441fbb26366244e2e101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d"
I20260929 19:02:44.312412 23143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [term 0 FOLLOWER]: Advancing to term 1
W20260929 19:02:44.312633 23200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:44.312685 23200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:44.313220 23143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f0ab0c996e441fbb26366244e2e101 in term 1.
I20260929 19:02:44.313441 22968 leader_election.cc:302] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530bccf2fc684bf6bd55c548dbf0aa3d, b1f0ab0c996e441fbb26366244e2e101; no voters:
I20260929 19:02:44.313556 23189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:44.313676 23189 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [term 1 LEADER]: Becoming Leader. State: Replica: b1f0ab0c996e441fbb26366244e2e101, State: Running, Role: LEADER
I20260929 19:02:44.313750 23189 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.314121 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f0ab0c996e441fbb26366244e2e101. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.314214 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:44.313153 22951 master_runner.cc:186] Error getting master registration for 127.0.98.61:40275: OK, instance_id { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" instance_seqno: 1790708564112354 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:02:44.315430 22951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:02:44.316444 23204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:44.316998 23204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:44.323403 23143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [term 1 FOLLOWER]: Refusing update from remote peer b1f0ab0c996e441fbb26366244e2e101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:44.323725 23189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:44.325069 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f0ab0c996e441fbb26366244e2e101. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.325179 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.325709 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.315435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.98.60:39077)
0929 19:02:44.315468 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.315470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.315785 (+ 315us) server_negotiation.cc:407] Connection header received
0929 19:02:44.316011 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.316012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.316024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.316054 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.316884 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.317604 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.318700 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.318984 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.319035 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.321344 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.321346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.321346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.321353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.323627 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.324443 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.324444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.324445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.324513 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.324837 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.324838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.324838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.324966 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.325039 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.325524 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.325601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:44.326767 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.326853 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:44.327168 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.327368 23212 mvcc.cc:205] Tried to move back new op lower bound from 7334742279462858752 to 7334742279429730304. Current Snapshot: MvccSnapshot[applied={T|T < 7334742279462858752}]
I20260929 19:02:44.327252 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.327751 23213 mvcc.cc:205] Tried to move back new op lower bound from 7334742279462858752 to 7334742279429730304. Current Snapshot: MvccSnapshot[applied={T|T < 7334742279462858752}]
I20260929 19:02:44.328205 23204 catalog_manager.cc:1347] Generated new cluster ID: 0599b25721bd453fb5dbd2f17893432e
I20260929 19:02:44.328255 23204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:44.328931 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.315912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.0.98.62:36837)
0929 19:02:44.315957 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.315959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.318963 (+ 3004us) server_negotiation.cc:407] Connection header received
0929 19:02:44.319105 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.319106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.319127 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.319156 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.319996 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.320707 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.321726 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.322017 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.322083 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.325595 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.325596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.325596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.325603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.327512 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.327917 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.327918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.327918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.327977 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.328222 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.328222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.328223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.328357 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.328450 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.328755 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.328829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:44.339013 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.316294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.0.98.61:40275)
0929 19:02:44.316333 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.316335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.323417 (+ 7082us) server_negotiation.cc:407] Connection header received
0929 19:02:44.323519 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.323520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.323531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.323561 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.324282 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.325010 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.326003 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.326301 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.326350 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.336273 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.336275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.336275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.336282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.337790 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.338129 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.338130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.338130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.338174 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.338331 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.338332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.338332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.338447 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.338520 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.338852 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.338920 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:44.348688 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } attempt: 3
I20260929 19:02:44.351370 23204 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:44.352063 23204 catalog_manager.cc:1504] Loading token signing keys...
Sep 29 19:02:44 dist-test-slave-gfsd krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:02:44.357662 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.348942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39077 (local address 127.0.0.1:52196)
0929 19:02:44.348987 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:44.348997 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:44.349031 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:44.349196 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:44.349197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:44.349376 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:44.349440 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.349442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.350391 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.350392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.351207 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:44.351209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.352526 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.352527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:44.352587 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.353052 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:44.353057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:44.354601 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:44.356480 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.356481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.356481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.356708 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.356951 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:44.356951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:44.356952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:44.357022 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:44.357335 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:44.357336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:44.357393 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 19:02:44.357452 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:02:44.357547 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:02:44.358260 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:44.348997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.60:39077)
0929 19:02:44.349035 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.349037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.349071 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:02:44.349092 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.349092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.349102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.349146 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.349522 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.350278 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.352075 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:44.352366 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:44.352429 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:44.354703 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:44.354704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:44.354705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:44.354712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:44.356370 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.356800 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.356801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.356802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.356869 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:44.357100 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:44.357101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:44.357101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:44.357238 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:44.357294 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:44.358085 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 19:02:44.358168 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:44.359014 23204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101: Generated new TSK 0
I20260929 19:02:44.359205 23204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:02:44.360106 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: Bootstrap starting.
I20260929 19:02:44.360836 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:44.361125 23094 log.cc:824] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:44.361944 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: No bootstrap required, opened a new log
I20260929 19:02:44.364084 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.364238 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c1fc0350314f28b019d388f379b68f, State: Initialized, Role: FOLLOWER
I20260929 19:02:44.364323 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } }
I20260929 19:02:44.364643 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:44.364835 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:44.365203 23218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.365358 23218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.365947 23067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [term 1 FOLLOWER]: Refusing update from remote peer b1f0ab0c996e441fbb26366244e2e101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:44.366128 23068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1f0ab0c996e441fbb26366244e2e101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77c1fc0350314f28b019d388f379b68f"
I20260929 19:02:44.366307 23189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:44.366986 23068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f0ab0c996e441fbb26366244e2e101 in term 1.
I20260929 19:02:44.368502 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.368629 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:02:44.368762 23233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:02:44.368820 23233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:02:44.368368 23218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f0ab0c996e441fbb26366244e2e101. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.368992 23218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:02:44.369911 23226 mvcc.cc:205] Tried to move back new op lower bound from 7334742279608918016 to 7334742279429730304. Current Snapshot: MvccSnapshot[applied={T|T < 7334742279462858752}]
W20260929 19:02:44.390049 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.388731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.0.98.62:36837)
0929 19:02:44.388772 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.388774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.389397 (+ 623us) server_negotiation.cc:407] Connection header received
0929 19:02:44.389510 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.389511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.389522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.389550 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.389984 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34872: BlockingRecv error: recv got EOF from 127.0.0.1:34872 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:02:44.390807 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.389109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.98.60:39077)
0929 19:02:44.389144 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.389146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.390286 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 19:02:44.390401 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.390402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.390412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.390441 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.390752 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52202: BlockingRecv error: recv got EOF from 127.0.0.1:52202 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:02:44.391789 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.391144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.0.98.61:40275)
0929 19:02:44.391185 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.391186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.391247 (+ 61us) server_negotiation.cc:407] Connection header received
0929 19:02:44.391407 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.391408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.391419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.391448 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.391732 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38344: BlockingRecv error: recv got EOF from 127.0.0.1:38344 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:02:44.393939 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.393208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.0.98.61:40275)
0929 19:02:44.393240 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.393242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.393475 (+ 233us) server_negotiation.cc:407] Connection header received
0929 19:02:44.393568 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.393569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.393585 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.393612 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.393899 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38350: BlockingRecv error: recv got EOF from 127.0.0.1:38350 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:02:44.396855 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.396087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.0.98.61:40275)
0929 19:02:44.396119 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.396121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.396369 (+ 248us) server_negotiation.cc:407] Connection header received
0929 19:02:44.396468 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.396469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.396479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.396508 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.396814 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38364: BlockingRecv error: recv got EOF from 127.0.0.1:38364 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:02:44.404413 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.403534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.0.98.61:40275)
0929 19:02:44.403570 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.403572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.403867 (+ 295us) server_negotiation.cc:407] Connection header received
0929 19:02:44.403969 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.403970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.403980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.404010 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.404369 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38366: BlockingRecv error: recv got EOF from 127.0.0.1:38366 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:02:44.410547 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.409738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.0.98.61:40275)
0929 19:02:44.409772 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.409773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.409999 (+ 226us) server_negotiation.cc:407] Connection header received
0929 19:02:44.410097 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.410098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.410108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.410136 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.410502 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38376: BlockingRecv error: recv got EOF from 127.0.0.1:38376 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:02:44.420686 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.419860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.0.98.61:40275)
0929 19:02:44.419899 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.419900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.420140 (+ 240us) server_negotiation.cc:407] Connection header received
0929 19:02:44.420252 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.420253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.420264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.420297 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.420637 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38378: BlockingRecv error: recv got EOF from 127.0.0.1:38378 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:02:44.438951 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.438010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.0.98.61:40275)
0929 19:02:44.438053 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.438055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.438339 (+ 284us) server_negotiation.cc:407] Connection header received
0929 19:02:44.438459 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.438460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.438474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.438504 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.438893 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38392: BlockingRecv error: recv got EOF from 127.0.0.1:38392 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:02:44.473100 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.472280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.0.98.61:40275)
0929 19:02:44.472328 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.472330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.472492 (+ 162us) server_negotiation.cc:407] Connection header received
0929 19:02:44.472565 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.472566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.472578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.472646 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.473025 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38408: BlockingRecv error: recv got EOF from 127.0.0.1:38408 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 19:02:44.539089 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.538239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.98.61:40275)
0929 19:02:44.538317 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.538319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.538443 (+ 124us) server_negotiation.cc:407] Connection header received
0929 19:02:44.538518 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.538519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.538532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.538608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.538995 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38422: BlockingRecv error: recv got EOF from 127.0.0.1:38422 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 19:02:44.669199 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.668363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.0.98.61:40275)
0929 19:02:44.668453 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.668455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.668555 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:02:44.668645 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.668646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.668665 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.668736 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.669120 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38428: BlockingRecv error: recv got EOF from 127.0.0.1:38428 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 19:02:44.927192 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:02:44.926440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.98.61:40275)
0929 19:02:44.926514 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:44.926516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:44.926532 (+ 16us) server_negotiation.cc:407] Connection header received
0929 19:02:44.926633 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:44.926634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:44.926649 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:44.926719 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:44.927066 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38430: BlockingRecv error: recv got EOF from 127.0.0.1:38430 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:02:44.940173 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1f0ab0c996e441fbb26366244e2e101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f0ab0c996e441fbb26366244e2e101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } peers { permanent_uuid: "77c1fc0350314f28b019d388f379b68f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40275 } } peers { permanent_uuid: "530bccf2fc684bf6bd55c548dbf0aa3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39077 } } }
I20260929 19:02:44.940343 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f0ab0c996e441fbb26366244e2e101 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:45.300263 23186 catalog_manager.cc:1259] Loaded cluster ID: 0599b25721bd453fb5dbd2f17893432e
I20260929 19:02:45.300354 23186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: loading cluster ID for follower catalog manager: success
I20260929 19:02:45.301255 23186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: acquiring CA information for follower catalog manager: success
I20260929 19:02:45.301602 23186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 530bccf2fc684bf6bd55c548dbf0aa3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:02:45.369262 23233 catalog_manager.cc:1259] Loaded cluster ID: 0599b25721bd453fb5dbd2f17893432e
I20260929 19:02:45.369340 23233 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: loading cluster ID for follower catalog manager: success
I20260929 19:02:45.370231 23233 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: acquiring CA information for follower catalog manager: success
I20260929 19:02:45.370575 23233 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77c1fc0350314f28b019d388f379b68f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:02:45.443225 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:45.440376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.98.61:40275)
0929 19:02:45.440532 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:45.440534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:45.440582 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:02:45.440711 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:45.440712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:45.440760 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:45.440842 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:45.441032 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.441656 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.442445 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.442667 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.442743 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:45.442882 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:45.442994 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:45.443055 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:45.443137 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 19:02:45.443156 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260929 19:02:45.447728 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:45.445207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.60:39077)
0929 19:02:45.445326 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:45.445328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:45.445337 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:45.445430 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:45.445430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:45.445482 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:45.445567 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:45.445683 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.446159 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.446905 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.447125 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.447197 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:45.447389 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:45.447492 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:45.447581 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:45.447620 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:02:45.447640 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260929 19:02:45.452185 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:45.449389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.98.61:40275)
0929 19:02:45.449442 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:45.449444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:45.449625 (+ 181us) server_negotiation.cc:407] Connection header received
0929 19:02:45.449688 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:45.449689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:45.449699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:45.449762 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:45.449900 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.450440 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.451378 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.451675 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.451735 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:45.451896 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:45.451930 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:45.451980 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:45.452080 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 19:02:45.452102 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:02:45.452463 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:45.449436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.98.60:39077)
0929 19:02:45.449473 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:45.449475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:45.449808 (+ 333us) server_negotiation.cc:407] Connection header received
0929 19:02:45.449863 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:45.449864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:45.449874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:45.449933 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:45.450043 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.450754 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.451666 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.451902 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.451967 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:45.452116 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:45.452147 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:45.452175 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:45.452380 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:02:45.452403 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:02:45.452960 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:45.449348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.62:36837)
0929 19:02:45.449492 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:45.449494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:45.449520 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:02:45.449630 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:45.449631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:45.449683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:45.449759 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:45.449950 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.450746 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.451728 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:45.452026 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:45.452087 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:45.452468 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:45.452732 (+ 264us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:45.452773 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:45.452875 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:02:45.452899 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260929 19:02:45.458348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 22951
I20260929 19:02:45.463666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 23026
I20260929 19:02:45.468376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHoo0t8/build/release/bin/kudu with pid 23101
2026-09-29T19:02:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2812 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5614 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271506 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST